Pretty cool to hear about your process - I've been thinking about this space a lot so have some context about pain points here.
I've found that building a benchmark is something that's easy-ish on the surface level, but has many layers of technical complexity the deep you go down the rabbit hole.
For example, a tangent on grading. Do you use test passing and/or LLM graders? If you're using an LLM, it's fairly easy to spin up a basic LLM grader. But - is that grader calibrated? Is it biasing towards same-model agents? How many results do you need to be above the noise floor? What is it even measuring, and is that aligned with what you care about? Does it have the right information to make those judgement calls?
I'm super curious how what you've been using this workflow for? Also happy to answer any questions / discuss further!
Yes, I'm with you 100% on the setup. You can go as deep as you want down the hole. At some point I had ask myself what I was really after with the benchmarks and where the line of diminishing returns was (and more importantly, was I really getting generalizable results).
I'm currently using a 2 tier grading system, one pass mechanical (LLM graded) one pass human. The grading ranges from -1 to 0 for the mechanical pass and -1 to 1 for the human pass. A model that does everything competently passed with a score of 0. Non-pass gets a -1. For the mechanical pass that's it. For the human pass there's a chance to get an extra point for exceptional work. Accepting the subjectivity in the human pass was part of the "what am I doing this for" point from above.
The scores then don't get read left-to-right and summed, but rather vertically per task and category. The temp setting as another commenter noted means you can't say anything definitive about the quality from a single run. What I think you can tell is 1) per task, who is the outlier in either direction and investigate, 2) how a task is generally handled by the models and 3) in the aggregate, who is dropping and gaining points consistently and does that change per task type. In the end, it's high touch. It doesn't give you one number to cut or keep a model. It tells you where to look and where not to.
You're the second person running an evaluation business to express interest, maybe I should start one as well ;) I will be reading your blog posts to see if there's anything I can adopt. If I was to give you the objection I would pose if this was an MBA brainstorming sessions I would ask:
Who are you targeting? and if the answer is engineers, what's to stop them from using the LLM to do it themselves? The problem space exists in the easy/hard domain; easy to set up, hard to refine because of nuance and specifics. Moreover, an engineer whose job might be under pressure because of LLMs would be highly motivated and highly capable to create a testing and evaluation setup and "own" this part of the process since it can't be replaced by an LLM.
Personally, I think this space is good consulting territory. There is hard-won knowledge that is valuable and sale-able, but beyond the basic setup, everything has to be tailored. It's also going to be hard to get people into a product moat. Git and IDEs give customers (if they are engineers) all the tools they need for this work and they already know how to use them.
One last note of your comment about same model agents. This is definitely something I have noticed. It's probably structural, but models absolutely prefer their own outputs to others. You can also move them toward agreement or disagreement by telling them that their model (in another session) produced some output. Kimi in particular will dunk on other models and promote what it thinks is its own output. Kimi is also highly sensitive to being bench marked. I have a specific audit script that goes through thinking traces to see if the model realizes it's being tested. Kimi figures it out more than any other model and will dig big time to find other results if you don't completely isolate it. Claude gets pissy if it thinks you are trying to benchmark it. Fable has inserted several times "not under testing" and "not a competitor" when left to its own committing. Fable is also very good at setting up open weights models and testing them. They must have spent a lot of time going for thoroughness and scientific method in training.
Artificial Analysis shows Grok 4.6 taking $1,068 to run their suite while Gemini 3.7 Flash takes $485. So it looks like Gemini 3.7 Flash is less than half the price in the real world.
Per-token cost isn't a great metric given that some use way more tokens than others.
Agree, with you but I'm still using 3.6 Flash because of tok/s/ latency/ uptime with high context. Tried Grok 4.6 and it was scoring lower on some internal benchmarks or slower.
> Hard to understand why anyone would choose 3.7 Flash under these conditions.. is Deepmind still a frontier lab?
At this point, I think they're mostly targeting Google One and Workspace subscribers, except doing worse compared to Microsoft because they don't have Microsoft's huge enterprise moat built from their DOS and Windows days.
This approach seems fundamentally predicated on being able to evaluate coding agents on your own code by having domain specific evals. With that knowledge, you can trust the routing logic is actually improving/maintaining perf while reducing costs.
Without the insight into agent performance, any changes like this feel like a gamble to save $$ at the cost of developer productivity
I'm actually working on building generic repo-specific benchmarks at https://stet.sh ;)
The difficulty of evaluating coding agents is indeed a really big challenge. We built evals on our own codebase and shared some information about that to allow other companies to replicate. We found our own evals correlated loosely with public generic SWE benchmarks.
In large user populations like at Databricks I think the ultimate answer will come from experimentation instead of offline evals. We are already doing this in small groups, exposing them to new candidate models and then measuring per-developer cost and perceived quality changes.
Experimentation makes a lot of sense as agents need to be pleasant to work with, and power users are a willing and able cohort to do that differentiation. Agent capability is only part of the equation, as evidenced by the widespread frustration with the experience of working with Opus 5, despite it topping benchmarks and being objectively extremely powerful/competent.
However, as a self-admitted eval true believer, I remain bullish that publicly available eval capabilities will catch up to current model intelligence, and start to be able to capture more and more of the subtle components of the agent experience, such as interactivity and code maintainability. Meta and Scale have some evals in the interactivity space that seem pretty promising https://arxiv.org/abs/2606.29957https://labs.scale.com/papers/swe-interact.
Human usability of the agent should definitely remain the top priority, but IMO we focus too much on the subjective, and not enough on the measurable/objective now, and I think/hope we'll see more companies move in the direction of Databricks
I’ve been thinking about how we practically implement this at an organizational layer for large codebases. There’s clearly alpha to be had in optimizing AGENTS.md / skills / tools / … to improve performance, quality, and cost efficiency of an agent. The problem is defining what quality means, and providing a way for agents to optimize the harness using that lever.
The first step I see towards this is building a generic, reliable, and accurate *fitness function* for codebases - turning PRs into gradeable tasks that an agent can solve, and improve on.
I’m pretty curious how others are optimizing the coding agent harness now, as this has been a huge pain point for my company (we’re pretty much relying solely on vibes).
I literally hold a "retro" with my agent after a session.
> Let's conclude with a retro. Did you run into any issues during this session that you think could be improved? Any failed tool calls, confusing docs/prompts, or tricky wording that took you effort to figure out, etc? Any final thoughts that you want to raise? Anything minor you didn't mention? Help make this codebase easier for the next agent to work in.
It has ideas for coding changes, spots unmentioned small bugs, suggests invariants (I have a high-level list of invariants to keep), principles to adopt, lint rules, tooling tweaks (I've built SO MUCH agent-tooling; so many useful bash scripts), skill-file updates, follow up work, all kinds of stuff.
Agreed. After an implementation session, I often use:
"did you notice any inconsistencies or opportunities to clarify/consolidate parameters or functionality while you were working?"
Another strategy that has worked well for my MCP server is noticing a sequence of failed tool-calls, then returning a response that contains a request for the agent to provide feedback via a skill.
I’m just holding out for when this stuff becomes mainstream and automatically built into tooling. I don’t want to hold a retro with my agent. I actually want to spend as little time as possible talking with it.
Sure, you can "vibe/auto-retro". But in my experience the retro often results in "higher-order" structural changes.
In starker terms: if you're ok with "vibe principles", "vibe invariants", "vibe lint-rules", "vibe skill updates", and "vibe agent-tooling"... well. That's your call.
Thankfully after a dozen or so retros the low hanging fruit starts to disappear. But until then, you've no idea what's there.
Ive been building my own ai voice agent harness from scratch for close to a year now and following good software architecture practices is a good start. So those rules have to be coded in agents.md somewhere also really helps to have a "vision" section or "spirit of the project" section that describes what the end goal vaguely looks like and things I care about in achieving for the project. This prevents agent from being brittle and "single minded" about its work. But yeah vibes most of the time is how I've also been doing it, but I did find one very important thing that has really sped up my work. So I figure I'd share it here. And that advise is to ignore front end design at all costs until the very end and you are ready to launch. UI related woes kill any type of inertia and are responsible for most of the issues. So I decided abandon all UI stuff until the very end and just focus on iterative refinement, cutting, and other back-end related work and its been great ever since. Ideas can be had, tested, validated/invalidates and you ' get stuck on the optimization thought loop. Recently I have started to consider how i can fully automate the development process as the capabilities are there but designing the instructions on how to do this well and how to handle niche cases without getting me involved takes careful planning in writing out the guide so that will be interesting to see once i get there.
Just working on my own stuff so YMMV, but from a cost reduction standpoint, the two most outsized ROI wins came from making sure the agent runs all terminal commands in quiet and using a codebase index mcp to speed up understanding and reduce full file reads [0]. I've also found the latter also dramatically improved the quality of output.
We actually had a lot of the tooling and process rules we now give out to LLMs so that they do what we want them to do optimally figured out and theoretically in place way before LLMs.
Turns out everyone had just been vibe–managing people this whole time so there was still that guy burning salary "tokens" going through a million lines of on–demand grep output.
(I'm the founder of Snyk and Tessl, apply what biases you wish)
I think we're mixing three optimizations: higher success rate, higher consistency and higher efficiency.
Success rate is about building confidence the agent can succeed. The hardest part here is defining what success is in the first place, and creating evals that help you measure it.
Consistency is about guardrails. The harness shines here, as it takes control away from the model, moving it to hooks that force certain paths, or require the use of deterministic tools for certain steps.
Given consistency and success rate, you can work on efficiency. You can save cognition (tokens) by moving work to tools, try to use a cheaper model, etc.
These three build on one another, at least if you want to scale them.
You can't improve consistency if you don't define success, and you can't add efficiency if you're not consistent.
It only tries to solve the „quality“ aspect of that equation tho, at least for the code output.
But I think for actual evals, some heuristics about a coding agent session are needed. But since the input tasks are always different, it’s hard to make any KPI actually comparable.
Did the agent needing 3 rounds of feedback on a big task perform worse than the one that needed none, but only worked on a small change?
You've just described what the AI industry refers to as "evals", a collection of which forms a particular benchmark. I definitely encourage you to define your own evals, because the public benchmarks are often either saturated (largely solved and only going to see small improvements going forward) or seemingly not predictive of real world performance. I could also go further and speculate that they are in the training data, although credible benchmarks avoid this, I'm just not sure how successful they are.
Thanks for the comment - I’ve actually been working on something to generate evals from private repos, as I 100% agree that public benchmarks are either contaminated (as OpenAI found with SWE-bench Verified), not high quality (as OpenAI found with SWE-bench Pro), or in the case of “good” benchmarks (Cognition’s Frontier Code comes to mind), non-representative of a specific repo’s work.
Sharing some challenges / learnings from the journey so far:
* Tests alone are non-representative: an agent can pass tests but write the code in a way that’s subjectively worse than another test-passing result
* However, tests are still important and the best form of
deterministic evaluation we have
* Creating executable environments to run the tests is challenging, especially when considering that this should be doable for arbitrary repos
* Selecting tasks that discriminate is an art - they need to be challenging, but not too challenging, represent the variety of work that’s done in the repo, and contain tests that failed prior to the change
* LLMs / agents are very powerful at judging code - but doing so in a way that is calibrated, consistent, and representative of codebase standards requires careful rubric creation and grading
* I’ve shifted towards using agents (from “static” LLM calls) to generate the rubrics, and to later grade the rubrics. This is powerful but even more non-deterministic
However what I wonder how we make these decisions on what to remove. Certainly some subset of the skills / AGENTS.md / prompt / tools do actually help agent performance, the problem is in identifying those, throwing away the rest, and validating that this didn't actually hurt the agent performance
Yes, my question is directed around how you are determining "task result." Is this measuring whether the code works, or whether it is maintainable?
As with human-powered coding, we read code far more frequently than we write it. It's worth spending a little extra (time|tokens) during authorship to make future maintenance feasible.
My CLAUDE.md, memories, and skills are all about either (a) adherence to project standards and guidelines, (b) product decisions which impact future code, and (c) instructions on how to prototype and work in my environment.
Removing these instructions would mean more turns with the AI to get the desired result.
You don't have to remove them - you can move things that are not important to every prompt to aseparate small docs that are referenced in agents.md with ,"when needed, reference these files:" and list them in form * relpath - content summary" or similar.
. When you want those instructions to be followed, mentiont them in your prompt. "Test this following procedures on docs/test.md". (I've found I don't have to do that extra instruction in al cases depending on model)
Aren't subdirectory specific Claude files layered in on top? Seems like more required content when the direction I'm suggesting is more like lazy-loading.
They load only when operating on files in that directory. i.e. Lazy loading?
For example, I have one in my schema migrations directory that explains how to write migrations with appropriate concern for forwards- and backwards-compatibility in our deployment model.
This type of content I use this for is not necessarily directory-specific - git operations as a simple example. I have workflows that are only relevant in specific situatoins, but they apply to the whole repository.
This is effective only if you're guiding it heavily though - this is a very human-supervised workflow, and I'm telling it when it's time to take specific actions. It will read the appropriate instructions file when the action is first referenced, but not until then.
in this case there was a hidden grader that checked if the implementation was correct (because that was the easiest thing to check), all 3 agents cleared this hurdle in all 9 runs
I agree, next it makes sense to try more open ended tasks + have humans (and/or multiple models) grade the runs and their results
> The problem is defining what quality means, and providing a way for agents to optimize the harness using that lever.
I've built and my team uses plannotator [1] for specs and code review. It stores all human annotated feedback, which we compile from time-to-time and iterate on coding standard skills for automated review (that seems to get better with the more feedback we provide).
You could write a script that does the same for your PR review comments, which it sounds like you maybe do with a bit more structure.
I have Opus grade each of my repos, and each subsystem of them, and do it every weekend.
Since the version of opus has changed over time, so have the evaluations. I was going to constrain it, but since they edit the same files, I can alter those (which it uses as summaries and starting points).
Also working on a product to build tasks from your own work for testing coding agents. Main thing I would offer is to look carefully at the agent trajectories - they love to figure out ways to cheat. Additionally, consider what "winning" means. If just using test pass rate, consider that tests might not encode what good means in your repo. I have been having success using "equivalence with merged PR" as judged by an LLM as a signal.
Yup, trying to be really strategic about testing. I didn't end up sticking with it, but I tried requiring test cases to cite a matching clause in the task assignment. But also: these tests are only indicative. Only a human can score a run.
To me, this comes down to verifiability. How do we measure the quality of what an agent is doing on our codebase rather than simply measuring task accomplishment?
> Verifying quality is orders of magnitude harder than "did the tests pass"
Agree that agentic grading is the future here. Cognition's Frontier Code is probably the best large public benchmark at this. You attribute agent quality issues to RLVR's binary pass/fail, however I wouldn't be surprised if labs are already supplementing that with rubrics as rewards to train more 'tasteful' models like Fable.
What can a practitioner do? I think there's promise in turning the optimization machine to the harness itself - building out a representative dataset of tasks on your repo, grading agent quality on them across various configurations, and optimizing [AGENTS.md / SKILLS.md / workflow / model / harness / tools] on that signal. High quality grading is still very hard, but it's more tractable at smaller, repo-level scale, and you can afford slower, more expensive verification for each task. You only need it to be right about your codebase's standards.
> In fact, it's not hard to imagine that if a model could reliably tell good code from bad, it might have written the good version to begin with
Pushing back slightly - detecting slop and discriminating quality is easier than generating it (why code review is so effective), and why grading is viable at repo eval scale even if it's much harder at RL scale.
Everyone is flying blind. For example, I am genuinely interested in trying HumanLayer, but would likely want some harder evidence (beyond anecdotes) that it's actually making my agents more effective before rolling out to an enterprise team.
I'm building this harness optimization loop @ https://stet.sh if curious
as a tip - models will always find a way to cheat, you will probably need to impose some restrictions on what they do / are able to access in the sandbox environment
It depends on what you're measuring. I agree that model resourcefulness is useful, but if you're trying to simulate real user sessions, then Claude looking at upstream Git and fetching the answer directly is somewhat worthless.
In my case, I'm trying to measure how coding agents perform under realistic scenarios when implementing tasks, as a proxy for how agents perform when used by actual users for those same tasks, so it's important to ensure the agents are behaving realistically instead of "cheating" and looking up answers.
Happy to share resources! I've been pretty deep in the space :)
- Mines tasks from your merged PRs/commits
- Replays them in Docker containers with different harness settings (change model / reasoning effort / AGENTS.md / etc)
- Grades the patches on various attributes (tests, equivalence with human patch, code quality)
The goal is to get a sense of how agents perform on your tasks, with your context, using the tools you do.
Thank you! Both of those papers are super new and super relevant.
The only big gap left is that they aren't using claude code/codex as harnesses. I'll try to reuse their constructed user sessions.
PS: Your work at stet is also interesting. That's definitely a problem right now that's hard to track. The only real solution is more robust CI/CD. I have since added harder validation like essentially running a full benchmark run on every prod push.
I've found that building a benchmark is something that's easy-ish on the surface level, but has many layers of technical complexity the deep you go down the rabbit hole.
For example, a tangent on grading. Do you use test passing and/or LLM graders? If you're using an LLM, it's fairly easy to spin up a basic LLM grader. But - is that grader calibrated? Is it biasing towards same-model agents? How many results do you need to be above the noise floor? What is it even measuring, and is that aligned with what you care about? Does it have the right information to make those judgement calls?
I'm super curious how what you've been using this workflow for? Also happy to answer any questions / discuss further!
reply