It is a neat tech demo but it clearly shows the limits of AI:
- I got it to generate invalid SQL resulting in errors
- it merely generates reasonable SQL, but in my case it generated to disjoint set of tables….
- In practice you have tot review all code
- It can point you into the wrong direction. Novel systems often have something smart/abstract in there. This system creates mostly
Straightforward simple systems. That’s not where the value is
All in all, it’s not worth it to me. Writing code myself is easier than having to review LLM code
Within our organization we have forbidden full LLM merge request because more often than not the code was suboptimal. And had sneaky bugs/mistakes.
I’m not saying these can’t be overcome. But not with current LLM design. They mostly generate stuff they have seen and are bad as truly new stuff.
I have had tremdendous success with using LLM's to generate SQL. In my use, the majority of the time, ChatGPT gets things spot-on. Even for really sophisticated queries that are going to inspect and aggregate multiple tables into one single output.
I do agree it is not perfect - and things need to be reviewed - but I rarely get the sort of gobbledygook that "resembles" valid SQL and is in fact meaningless.
meaningless is good. You can take one look and discard it. The problem is that you can't trust there aren't subtle errors though. So you still need to go everything with a fine toothed comb. If you don't, you're just sitting on a ticking timebomb. "it seems to be working" is very very different to "this does what I had in mind and it will work for all inputs"
- I got it to generate invalid SQL resulting in errors - it merely generates reasonable SQL, but in my case it generated to disjoint set of tables…. - In practice you have tot review all code - It can point you into the wrong direction. Novel systems often have something smart/abstract in there. This system creates mostly Straightforward simple systems. That’s not where the value is
All in all, it’s not worth it to me. Writing code myself is easier than having to review LLM code
Within our organization we have forbidden full LLM merge request because more often than not the code was suboptimal. And had sneaky bugs/mistakes.
I’m not saying these can’t be overcome. But not with current LLM design. They mostly generate stuff they have seen and are bad as truly new stuff.