OK, then the current models aren't as good as I thought/hoped.

I guess one thing it means is that we still need extensive test suites. I suppose an LLM can write those too.