Sorry for the inconvenience. We ran 27,280 total trials on Runta (1–4 vCPU, 2–8 GB RAM, 32 GB disk) using the 62-task [email protected] dataset across 5 models and 22 harnesses on Baseten, with 4 attempts per task. Every harness and model ran at max effort with a 131,072 max output token limit per request and per-task agent timeouts ranging from 15 minutes to 8 hours.
For scoring, final passes are divided by scored attempts, where infrastructure errors are excluded from the denominator, timeouts are still graded by the verifier, and reward hacking (such as looking up solutions online) counts as a failure. Costs are calculated using list price multiplied by reported tokens.
These kind of benchmarks declare "best harness" but measure just individual tasks with zero human interactivity, not project level efficiency. I don't do individual tasks, I work on larger projects made of 100s of tasks. The work across tasks, keeping intent alignment straight and testing reliable, remembering that detail from 5 days ago, reducing repetitive friction by fixing skills and tools, that is also important.
This pretended equivalence is like claiming good LeetCode score means good engineering. I know it is much harder to measure project-wide performance but that is what we need.
unless they edited it in later, the link at the bottom lists which tasks they ran it on. It definitely doesn't answer why those tasks or why specifically 62 of them
We just want to keep the category unified so we have tasks for every domain like ML, science and others, and also have a chance to see the divergence from tb2.1 to tb4.
reply