Hacker Newsnew | past | comments | ask | show | jobs | submit | sjmaplesec's commentslogin

We ran a bunch (2,725) of Tessl verifier tests which are part of our test suite for our internal code base and switched the judge model to Jev and GPT Luna 6

The results:

- Jev is 13.6x faster - Jev is 2.7x cheaper - The models had a 85.9% agreement on verdicts

Jev finished the suite in 32 seconds. GPT Luna 6 took 436.5 seconds.

The interesting part is where the models disagreed, particularly on rules about comment structure and content. We published the approach, what we tested, and a workflow you can reproduce on your own codebase.


Full set of models tested: claude-opus-4-7 claude-opus-4-6 claude-sonnet-4-6 claude-haiku-4-5 gpt-5.4 gpt-5.3-codex gpt-5-codex cursor-composer-2

11 Skills used were here: https://github.com/mcollina/skills


Pass in a skill and it'll roast the contents:

"This is not just useless - it is an insult to the very concept of functionality."


I ran some evals to see which Anthropic models use skills the best, between Opus, Sonnet and Haiku.

I was pretty impressed how good Haiku was with skills at completing various tasks


Link to all the review scans is here - mostly in the 50-70% range https://tessl.io/registry/skills/github/googleworkspace/cli


There's so much more we can do around activation and skills creation. Looking at the eval results, there are even cases where the context makes the agent worse.

Scenario 5, test 1 72% -> 22%

https://tessl.io/eval-runs/019cc02f-bb26-76e0-a7c9-598a7337e...


The review eval tests language, activation etc of skills. I guess you could move it all to a skill quick and then run an eval on that if using Tessl. This checks if the way you write the instructions etc are being well understood by the agent


An eval is to an LLM as a test is to code.


Tessl can generate the evals, both to test anthropic best practices as well as running scenarios with and without the skill to check if it's helping


Can add this as a skill or as part of a skill, and so you don't need to keep prompting the same things.


Consider applying for YC's Winter 2027 batch! Applications are open till November 2.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: