Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Just by the (lack of) inter-model variance, I don't think SWEBench-Pro does a very good job of representing model capability. Terminal-Bench seems more challenging and separates the wheat from the chaff.

Also, *ops work, which in my experience can actually be more complicated than SWE is underrepresented there obviously.



Consider applying for YC's Winter 2027 batch! Applications are open till November 2.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: