Hacker Newsnew | past | comments | ask | show | jobs | submit | Edward40's commentslogin

I don't mean the harness, just the loop. Memory could be a part of the harness.

Sorry, I misused OP there, was referring to @cpeth in this comment thread.

Thanks for your eval work, it seems like a tough area to measure!


Kilo does not seem that popular. Point it out if I am wrong.

I think Claude Code does not care about token efficiency, the more users use, the more users pay.

We ran 4 attempts per task, that's just like when you roll dice 4 times.

Although people disagree with the best models, the worst model is obvious.

I am thinking about how to measure the efficiency of ultilzing existing context in one specific domain.

Good finding! Model preference is interesting.

I'm also building a leaderboard in the meantime, which will release in a few days with more detailed harness version info and methodology.

Sorry for the inconvenience. We ran 27,280 total trials on Runta (1–4 vCPU, 2–8 GB RAM, 32 GB disk) using the 62-task [email protected] dataset across 5 models and 22 harnesses on Baseten, with 4 attempts per task. Every harness and model ran at max effort with a 131,072 max output token limit per request and per-task agent timeouts ranging from 15 minutes to 8 hours.

For scoring, final passes are divided by scored attempts, where infrastructure errors are excluded from the denominator, timeouts are still graded by the verifier, and reward hacking (such as looking up solutions online) counts as a failure. Costs are calculated using list price multiplied by reported tokens.

https://hub.harborframework.com/datasets/frontier-harness-ev...


These kind of benchmarks declare "best harness" but measure just individual tasks with zero human interactivity, not project level efficiency. I don't do individual tasks, I work on larger projects made of 100s of tasks. The work across tasks, keeping intent alignment straight and testing reliable, remembering that detail from 5 days ago, reducing repetitive friction by fixing skills and tools, that is also important.

This pretended equivalence is like claiming good LeetCode score means good engineering. I know it is much harder to measure project-wide performance but that is what we need.


This doesn't answer the question at all.

unless they edited it in later, the link at the bottom lists which tasks they ran it on. It definitely doesn't answer why those tasks or why specifically 62 of them

We just want to keep the category unified so we have tasks for every domain like ML, science and others, and also have a chance to see the divergence from tb2.1 to tb4.

It's still the most informative thing to come out of that account or the linked article.

Cutting the system prompt and tools is easy. Cutting steps requires understanding the model's RL strategy and how it was trained to explore.

I wonder if Claude Code would do better given they just introduced code mode, which should significantly cut the number of steps required

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: