Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

While in general that is true, this is one person and so we can assume a lot of the variables that make it a poor metric are constant in this case and so we can draw a few conclusions.
 help



From my own experience, LLM generated code having more test cases is neither surprising nor an indication of improved quality. I've found LLMs are quite good at making lots of test but the tests are often not that useful. I would argue that total count remains a poor metric for quality.

I don't know of a good metric for quality. Total count of tests is the best I've found, but I agree it is bad.

Best metric is user satisfaction. Since that is ultimately what software is (or should be) about. Hence the leading header of "software is getting worse for obvious reason": we detached from user satisfaction as the metric to optimize for at least a decade ago.

Ask for red-green tests. Without that they are often indeed useless.

I'm curious, mind to tell me more about your process?

Just simply ask for that, and they usually know what to do.

They should write a test and stash the fix they just wrote. In this state it should fail. Once they re-apply the patch, it should become green and this gave use enough information to deduce that the test does test something.


We can conclude that there are more tests per code perhaps. We can't conclude they're adding value.

my experience is 25% of valuable. Which is a lot better than nothing.



Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: