If anyone wants this kind of benchmark for their own codebase, happy to set you up with superconductor.com/benchmark
1. Import your own PRs
2. We find the original spec or infer the spec
3. Agents you select (eg claude code opus 5, codex gpt 6 astra high, pi kimi k3, etc) implement the spec (starting from the parent commit of the PR, with the git history is pruned so they can’t look up the implementation)
4. Three judge LLMs grade agent solution given spec, PR implementation, and a rubric
5. You see the scores
Check out superconductor.dev (I’m building it), if you want live app previews, docker-in-docker functionality, multiple agents in one mobile app, and more.
> AFAICT, there’s no service that lets me: give a prompt, write the code, spin up all this infra, run Playwright, handle database migrations, and let me manually poke at the system. We approximate this with GitHub Actions, but that doesn’t help with manual verification or DB work.
- set up your environment any way you want, including using docker containers
- run any number of Claude Code, Codex, Gemini, Amp, or OpenCode agents on a prompt, or "ticket" (we can add Cursor CLI also)
- each ticket implementation has a fully running "app preview", which you can use just like you use your locally running setup. your running web app is even shown in a pane right next to chat and diff
- chat with the agent inside of a ticket implementation, and when you're happy, submit to github
(agents can even take screenshots)
happy to onboard you if that sounds interesting, just ping me at [email protected]
My org has built internal tooling that approximates this. It's incredibly valuable from a manual test perspective though we haven't managed to get the agent part working well, app startup times (10+ min) make iterating hard.
Do you have customers who have faced/solved this problem? If so, how did they do it -- it seems like a killer on the approach?
Our foundational design value was compute instance startup speed. We've made some design decisions and evaluated several "neocloud" providers with this goal in mind.
Currently, from launching an agent to that agent being able to run tests in our Rails docker-compose environment (and to the live app preview running), is about 30 seconds. If that agent finishes their work and goes to sleep, and then hours later you come back to send a message, it'll wake up in about the same time.
(And, of course, you can launch many agents at once -- they're all going to be ready at roughly the same time.)
will email! Your homepage doesn't make the environment part clear - it reads like it's akin to cursor multiple agent mode (Which I think you had first, FWIW).
This is exactly why we built superconductor.dev, which has live app preview for each agent. We support Claude Code as well as Gemini, Codex, Amp. If you want to check it out just mention HN in your signup form and I’ll prioritize you :)
Not yet! We haven't shared them publicly yet because our internal dataset is super biased. Keep your eyes peeled though! They'll be coming out in the next few weeks :)
If you find yourself in Madrid, you will not regret making time for the Prado. Whereas some museums, like the Hermitage, go for quantity, the Prado seems to pride itself on extreme curation. It did not feel like a huge museum (at least the tour of it that I did), but every single room had at least one true masterpiece.
I agree. Do also visit the Reina Sofia[1], mostly dedicated to (modern) Spanish art.
It's your chance to experience Picasso's Guernica[2] in real. To me it was one of the most moving and impressive, while also disturbing exhibits I've ever seen in any museum.
1. Import your own PRs 2. We find the original spec or infer the spec 3. Agents you select (eg claude code opus 5, codex gpt 6 astra high, pi kimi k3, etc) implement the spec (starting from the parent commit of the PR, with the git history is pruned so they can’t look up the implementation) 4. Three judge LLMs grade agent solution given spec, PR implementation, and a rubric 5. You see the scores