The point is that it's trivial to fix the problem of cooking the waterfowl or writing your code with existing tools. Just ask them to verify.

If you have to say "don't make up something" for every possible question you ask in order for it not to make up something, that's a massive usability issue for regular people. If saying "don't make up something" will still result in it making up something up some of the time, that also might be a massive usability issue depending on if "some of the time" means 0.0001% or 1%.

Having a natural language interface where you need to go out of your way to specify that you want an accurate answer rather than just a plausible one defeats the entire purpose of it being a natural language interface for normal people. In certain professional contexts, it can be useful, but I don't buy it at all that it makes sense to ask everyone in their everyday lives to go out of their way to specify that they actually want correct answers to their questions.

> If you have to say "don't make up something" for every possible question you ask

You don't. It goes in the system prompt.

In practice it doesn't though, for any actual products on the market today.

Put it in your own system prompt. They all provide tools for doing this. OpenaAI Custom GPTs, Gemini Gems, Claude Projects, etc. Or you can easily roll your own using their APIs.