Happy to — please open an issue at https://github.com/fstandhartinger/jevbench/issues with an endpoint or runnable code, the model and licence, and whether it was trained on the public items. Every entrant runs through the same harness, including the sealed set. I definitely prefer open models I can run locally, because sending our private held-out set of tasks to an external API produces some headaches on ur side and basically means we have to rotate the test set a lot, to avoid contaimination. We can call API hosted models though, we'll just flag the leaderboard entries appropriately.

Thanks! Yeah, the code is open: https://github.com/fstandhartinger/who-is-right. Email classifying is definitely a good Jev usecase, I agree

Agree the coverage question is real. Today it's text decisions (intent, routing, extraction, scoring, harder multi-constraint cases). We're working on screen/computer-use and image decisions (a noindex preview exists) because that's closer to where Jev-class models will actually be used. Suggestions for task families are very welcome as GitHub issues.

Good call... BART-large-MNLI is the classic zero-shot baseline and it should be there. Adding it to the next run.

maybe open source even eats Jevs lunch first

see here: https://news.ycombinator.com/item?id=49800574