Which evals would you be interested in specifically? And what are you missing in the readme?

valid question. I thought a lot about this. The prefix before every fold stays intact (cache hit). But for the messages after the fold there is always a cache miss. Folds don't happen too often, so in practice this turned out to be worth it, because the cache miss comes with a big token reduction in general. I don't have a solid cost analysis for this yet, though.

Yes, you need an extension that uses the claude code credentials from the file system, like this: https://github.com/fdietze/pi-claude-auth

Works pretty well for me, even with latest opus-5-5

We probably want to compare the cartesian product of model+harness.

Agreed. Which other software does society need badly?

What matters more is if there is good QA.

Also, you can have NixOS instead of debian: https://github.com/nix-community/nixos-avf

Wouldn't it make sense to have these wikipedias created by different llms, and compare them to expose their biases?

It's not only the amount of code but also the quality of the available code. If a language has a low barrier to entry (e.g. python, javascript), there will be a lot of beginner code. If a language has good static analysis and type checking, the available code is free of certain error classes (e.g. Rust, Scala, Haskell).

I see that difference in llm generated code when switching languages. Generated rust code has a much higher quality than python code for example.

I fully agree with this approach! 5 years ago I built a prototype to execute that same concept of language covering. But instead of just using words, I used n-grams. It ii trained on subtitles to model spoken language. Combined with sqlite in the browser to get the next sentence with the most impact.

github here: https://github.com/fdietze/ravioli

prototype deployed here: https://raviolio.web.app/