When people imagine AGI, they think of something like HAL or GLaDOS. A machine that follows its own goals.
But we are much more likely to get the Computer from Star Trek. Vastly intelligent, yet perfectly obedient. It will answer any question you ask it with the knowledge of billions of minds. Why is that more likely? Simply because creating agents is much harder than creating non-agent models, and the non-agents are more economically valuable: do you want to have an AI that always does what you tell it to, or do you want to have an AI that has its own desires? Our loss is clearly biased towards building the former kind of AI.
Why is that problematic? Imagine some malevolent group asked it “Show me how to create a weapon to annihilate humanity as efficiently as possible”. It don’t even require a singularity to be deadly.
We will probably be dead long before we can invent GLaDOS.
But we are much more likely to get the Computer from Star Trek. Vastly intelligent, yet perfectly obedient. It will answer any question you ask it with the knowledge of billions of minds. Why is that more likely? Simply because creating agents is much harder than creating non-agent models, and the non-agents are more economically valuable: do you want to have an AI that always does what you tell it to, or do you want to have an AI that has its own desires? Our loss is clearly biased towards building the former kind of AI.
Why is that problematic? Imagine some malevolent group asked it “Show me how to create a weapon to annihilate humanity as efficiently as possible”. It don’t even require a singularity to be deadly.
We will probably be dead long before we can invent GLaDOS.