Hi HN, I’m Guanlan, founder of Runta. The eval runs on our runtime, so weigh this accordingly.
The short version: when you change the model, think about changing the harness too. On any one model, cost per pass varies by up to 4.4x
What we ran: 22 harness configurations × 5 open-weight models × 62 terminal tasks × 4 attempts. 27,280 runs, 83B tokens. Each harness ran with default settings at maximum reasoning effort, each trial in a fresh sandbox. Costs are Baseten list prices times reported tokens.
What we found:
The lowest-cost harness changes with the model. Codex CLI ranks 1st on cost with GLM-5.3 and 15th with Kimi-K3.
Paying more doesn’t buy passes. The harness with the highest pass rate ranked anywhere from 4th to 19th on cost.
The model still sets most of the bill (93% of cost variation). The harness is the second lever.
Starting prompt size doesn’t predict cost. Output tokens do.
64% of inference spend went to runs that didn’t pass.
Hi Guanlan, I vouched for your comment since you involved in this post, but you should be aware that all of your posts have been flagged, likely for AI patterns in your writing.
> Don't post generated text or AI-edited text. HN is for conversation between humans.
This is why for open-weights models at least harness aware post-training is becoming a thing. Also closed source harnesses that aren't fully editable are basically an anti-pattern. The labs should just hurry up and productionize self-evolving harnesses, and have the models internalize harness engineering, so they can optimise the harness for the given task. These are solvable problems.
This. I have coworkers who are really smug about claude code being "the best" experience for AI assisted development and who refuse to try anything else. They also constantly submit pull requests littered with trash "trace of thinking" slop comments and when I point it out they say "yeah I usually try to make it not do that but this one got through". Meanwhile in pi this was trivial to fix and I'm often mining session history to refine the efficiency.
The short version: when you change the model, think about changing the harness too. On any one model, cost per pass varies by up to 4.4x
What we ran: 22 harness configurations × 5 open-weight models × 62 terminal tasks × 4 attempts. 27,280 runs, 83B tokens. Each harness ran with default settings at maximum reasoning effort, each trial in a fresh sandbox. Costs are Baseten list prices times reported tokens.
What we found: The lowest-cost harness changes with the model. Codex CLI ranks 1st on cost with GLM-5.3 and 15th with Kimi-K3. Paying more doesn’t buy passes. The harness with the highest pass rate ranked anywhere from 4th to 19th on cost. The model still sets most of the bill (93% of cost variation). The harness is the second lever. Starting prompt size doesn’t predict cost. Output tokens do. 64% of inference spend went to runs that didn’t pass.
reply