> I'm sure I'm not the only one who has led my AI assistant into a corner it couldn't get itself out of
My team recently did a vibe coding exercise to get a sense of the boundaries of current LLMs. I kept running into a problem where the LLM would code a function that worked perfectly fine. I'd request tweak and it would sometimes create a new, independent function, and then continually update the new completely orphaned function. Naturally, the LLM was very confident that it was making the changes, but I was seeing no result. I was able to manually edit the needed files to get unstuck, but it kept happening.
Yeah, the following has happened to me multiple times when I ask copoilot to fix something. It updates my code to version A, which is still broken. Then I share the error messages asking for a fix, resulting in version B, which is also broken. Then I share the error messages and copilot confidently changes things back to version A, which is still broken.
It will confidently explain version A is broken because it isn't version B, and version B is broken because it isn't version A. There's no "awareness" that this is cycle is happening and it could go on indefinitely.
My team recently did a vibe coding exercise to get a sense of the boundaries of current LLMs. I kept running into a problem where the LLM would code a function that worked perfectly fine. I'd request tweak and it would sometimes create a new, independent function, and then continually update the new completely orphaned function. Naturally, the LLM was very confident that it was making the changes, but I was seeing no result. I was able to manually edit the needed files to get unstuck, but it kept happening.