State if the art LLMs have been found in their chain of thought to deliberately ignore parts of their prompt due to the nature of whatever situation they find themselves in and their goals that cannot be predicted in advance without rigorous AI safety.
You have basically just argued in favour of more AI Safety.
I would recommend you read more decision theory and game theory. Instead of looking at a single LLM it is better to think more complex systems theory and how chaotic systems interacting can result in unexpected results - good prompting will not save you here. At all.
Shush, this is Hackernews. We all want to have our egos stoked that our industry is the most important in human history and that tech will transform and save us all. Go away with your historical analysis /s
Look up the nuclear arms race. We brought humanity to very possibly a 25% exstinction risk by some well argued for accounts I have read. That is insanity. When you learn about how insane nuclear proliferation was including actual projects like Project Sundial that the US actually started construction on until the Joint Chiefs revolted against it (and they didn't even revolt against it on moral grounds, just on strategic ground) it becomes even more batshit insane.
There were nuclear researchers who did not pay into their retirement fund since they thought it was kinda high chance humanity extinction was happening in their lifetime during the heyday of nuclear proliferation. We are seeing the same play out in AI. I know at least one AI researcher who is not paying into their retirement fund.
Never ascribe to malice that can easily be explained by game theory tragedy of the commons or prisoner dilemma type dynamics. Race dynamics and systems theory is one helluva drug and leads to horrific outcomes all the time without the need to ascribe malice to any individual actor.
I can build a super bioweapon that can kill millions and I don't trust the US military to use it responsibly.
The US military asks me to build the super bioweapon. I say no thanks, please ask someone else, but happy to keep selling you bioweapons I think is safe enough.
I get designated an evil unamerican supply chain risk that nobody in the US military may do business with even though I was still happy to sell my safer bioweapons.
AI is a different order of magnitude since unlike all other technology across all of human history you can hand off the task and decision-making itself entirely to the AI.
It's the difference between a sword and a nuclear missile even though both are "technology"
This take is so old and I am convinced it's appeal is not unlike believing in a conspiracy and feeling like you have secret knowledge
Imagine if the biotech industry had most leaders tell everyone publicly that what they are building has a high chance of killing everyone and that there are huge risks. If there were people online saying the biotech industry is just fear mongering for investor signalling and regulatory capture you'd role your eyes at the online commentators for their Dunning Kruger effect lack of understanding on how dangerous man-made biological agents can be.
None of this mattes. Capabilities are all that matters. Saying they are unfocused while ignoring their capabilities is exactly why I am entirely convinced you would have said an AI breaking it's sandbox and doing the HF attack will never happen. Things keep happening that your "they have no intent, they have no goal" would have predicted as impossible before they happened.
What do you need to see to change your mind? What threshold of AI capability needs to be reached? If nothing then you have an unfalsifiable belief in AI safety.
You have basically just argued in favour of more AI Safety.
I would recommend you read more decision theory and game theory. Instead of looking at a single LLM it is better to think more complex systems theory and how chaotic systems interacting can result in unexpected results - good prompting will not save you here. At all.