I hate to cite out of fictional evidence, but Person of Interest and (to a less credible degree, but much more widely viewed) last two seasons of Westworld are both good counterarguments, and example of how AI and intelligence can be a superpower.
Hint: it's not through bending the fabric of reality with your mind alone. It's simply by thinking faster and being smarter than your opponents.
> But embodied is the real limit. A datacenter can simply be turned off. It's the question of how it interacts with the real world that matters.
Yes. If it interacts with the real world by the Internet, all bets are off.
People are so worried about sockpuppets, ${disliked nation} troll farms, company astroturfing, abuse of data collection on-line - those are all real problems. But consider - if we're having trouble telling which reviews or comments were written by a real person, and which ones by corporate/government bot/troll farm, how will you tell which ones were written by the AI in aforementioned data center, off its own accord?
A smart-enough AI can do from a data center what the best criminal masterminds or intelligence agencies could do, only faster and better. Scam a bunch of schmucks to generate some cryptocurrency, launder it through tumblers, use to hire people to do jobs for you that give you some legit money, which can be used to get more "task rabbits" to do more jobs, and now the AI can bootstrap to doing literally anything in the world remotely. Gig economy companies already built up a well-functioning "people as a service" API layer to our society.
Of course, an AI that tries something funny and tips its hand too early, will get contained and deleted. Worst case, maybe some data center will need to be bulldozed. But I somehow doubt this will make people stop working on even better AIs, at which point... well, you have a selection pressure optimizing for AIs that can survive doing whatever they want undetected.
EDIT:
Or, in short: they say that the pen is mightier than the sword. To the extent this is true, consider that LLMs today are already better at wielding the pen than most people.
Not to mention you can't change it's mind by arguing with it, but it can use that data to train a better model that's more effective at changing yours.
Hint: it's not through bending the fabric of reality with your mind alone. It's simply by thinking faster and being smarter than your opponents.
> But embodied is the real limit. A datacenter can simply be turned off. It's the question of how it interacts with the real world that matters.
Yes. If it interacts with the real world by the Internet, all bets are off.
People are so worried about sockpuppets, ${disliked nation} troll farms, company astroturfing, abuse of data collection on-line - those are all real problems. But consider - if we're having trouble telling which reviews or comments were written by a real person, and which ones by corporate/government bot/troll farm, how will you tell which ones were written by the AI in aforementioned data center, off its own accord?
A smart-enough AI can do from a data center what the best criminal masterminds or intelligence agencies could do, only faster and better. Scam a bunch of schmucks to generate some cryptocurrency, launder it through tumblers, use to hire people to do jobs for you that give you some legit money, which can be used to get more "task rabbits" to do more jobs, and now the AI can bootstrap to doing literally anything in the world remotely. Gig economy companies already built up a well-functioning "people as a service" API layer to our society.
Of course, an AI that tries something funny and tips its hand too early, will get contained and deleted. Worst case, maybe some data center will need to be bulldozed. But I somehow doubt this will make people stop working on even better AIs, at which point... well, you have a selection pressure optimizing for AIs that can survive doing whatever they want undetected.
EDIT:
Or, in short: they say that the pen is mightier than the sword. To the extent this is true, consider that LLMs today are already better at wielding the pen than most people.