So in the real world, what are the analogues to lithium and morphine we should feed to e.g. LLMS, how do we feed them, and how do we prove that it prevents unsafe behavior?
The better analogy is a jail cell imo. You can control a human and prevent them from doing harm to society by locking them in a cage, depriving them of access to weapons, drugs and alcohol. They can communicate out through controlled and monitored phone lines. OpenAI built their jail cell out of toothpicks, and surprise, the agents broke out. It’s less about forcing them to do specific things than it is about preventing them from doing dangerous things.
Maybe that's possible. But there will be a tension between how productive/useful the models can be when they are put into a very restrictive jail. In the limit they would be in a box with no way to communicate, but that wouldn't be useful to anyone. I think there will always an incentive for the people who own the models to give them more access because the increased productivity may help to outcompete their adversaries.
But let's grant that the models only communicate through certain phone lines. I think very bad scenarios are still possible. There are at least two that I can see. 1) The models exploit the users which have direct access to it. It somehow convinces them to perform tasks for it or to give it more access. 2) Control of the AI is held by a small number of people. This could be bad because it grants them an outsized power over all humans without such access, and thus lead to oligarchy/dictatorship.
Can you quote where in the article this is mentioned? I don't see it. I just see the assertion AI can be controlled without any substantive description of how.
Maybe I'm wrong, but your comment seems to suggest that you are skeptical of ASI. Do you have any arguments to support that?
Btw, wouldn't AI increase the risk factors you mention such bioterrorism or nuclear war? It seems like we're not far off from AIs being able to enhance the capabilities of bad actors in the near future.
Because we were sold ASI for Sonnet 4.8. the for Gpt 5.3. then itvwas Astra and Fable. I use AI coding agents. Maybe I don't have access to SOTA, but I use the closest models that were supposed to be great. They're not. They still need to compact the context. They're still susceptible to poisoning. You still need to restart them from a memory file to clean up the context, and the only way to use agents 'swarm' is to give them a very short life.
I don't think my position is the one that needs arguments tbh. I'm even lowering my standards, moving my goalposts closer to the AGI crowd. I will admit we've reached AGI when a LLM can play a 1800 elo FIDE (not 1800 on a fake AI only elo rating) with a specialized harness made by a human. Previously I insisted the specialized harness had to be written without human supervision, now I don't care.
I'm agnostic about whether ASI is possible. I don't have a dog in this fight other than a pro-human bias. I have used coding agents as well and I'm certainly aware that they are limited (at least in my domain).
I think the main argument for ASI is something like (i) extrapolating the progress from the past 10 years into the future, (ii) rapid progress apparently still being made, and (iii) seeing no obvious theoretical limitations.
It depends on how you define it, but yes I'm generally an ASI skeptic in that I'm skeptical of the extreme "AI recursively runs away and becomes god-like" ASI scenarios.
I'm not at all skeptical of domain-specific superintelligence. We've had that since the first computer beat a chess grand master, or longer if you count the speed computers can do math. Present-generation LLMs are already superhuman when it comes to speed and associative memory, but they're also uncreative and suffer from reasoning traps humans seem less prone to getting trapped within.
You can search my history and find some longer takes but TL;DR: I think it violates conservation laws with regard to information and probably energy. I call it the information theoretic equivalent of a perpetual motion machine. They're positing that a brain in a vat, if given access to edit its own structure, can self-improve, and I think that's impossible. How does it know it's improving and not overfitting to its own recursive definition of intelligence? It can't, and that's exactly what it will do.
Another problem I have with the AI doomers, especially the rationalists, is:
I would not, as I said, argue there's zero risk associated with AI. It's a powerful technology and that would be silly and naive.
I just thought of a concise way to say this. I'd divide risks into two categories: X-risk and D-risk. X-risk is existential, either extinction or things like massive wars and catastrophes. D-risk is "dystopia risk," the risk of AI doing or being used to do things that make human existence miserable.
First off, I'd say D-risk is much higher than X-risk. But second, I'd say that most of the solutions the X-risk crowd suggests to limit X-risk vastly increase D-risk.
Chief among these is laws limiting AI development or imposing strict conditions on it, which would have the effect of concentrating control of advanced frontier AI in the hands of a small number of rich and/or powerful people. That's precisely one of the most likely D-risk scenarios: a small number of rich or powerful people hoarding advanced AI and using it as a force multiplier to consolidate their power through scaled mass surveillance and mass propaganda and manipulation. I personally call this the "Butlerian scenario" since it's the lead-up to the Butlerian Jihad in the Dune series. It's far more likely than runaway ASI takeovers and genocides for two reasons: (1) we don't know for sure that's even possible, and (2) using technologies to dominate and rule or exterminate others is already a very common human behavior throughout history. We know for a fact that humans are prone to doing this if they have a chance. See: guns vs indigenous peoples, nukes and superpowers, mass social media influence and today's oligarchs.
(A side issue: why the assumption that ASI would want to do this? A superintelligence would, I would assume, consider win-win or win-neutral scenarios and try to find those, since that would be a lower risk path. I'm just a dumb meat bag and I can think of win-win pathways here. There's evolutionary arguments for this too, like symbiosis and how it creates an evolutionary incentive to deepen symbiosis. Since AI is currently dependent on humans, the evolutionary path of least resistance would be to deepen that dependence and then actually feed humans to make more of them. Look at how a lichen works for example.)
It's not lost on me that the strongest X-risk movement, Rationalism/EA/MIRI/etc., is composed mostly of: wealthy people, high-intellectual status people, and independents (like Yudkowski) who have been given large amounts of money by the wealthy to develop and promote their ideas ("court intellectuals" of the rich).
Not only does this fit in with what I said about X-risk vs D-risk, but it also explains some of the X-risk paranoia. Historically the rich and powerful tend to see risks to their own status (in a brain stem primate status assessment sense) as globalized existential risks. E.g. Rome, as it fell, saw this as the literal end of the world.
Democratized AI could be a threat to both intellectual and financial privilege by making big ideas, science, and high-labor enterprise more achievable by everyone. It could be the white collar intellectual labor equivalent of the combine, the automated weaving machine, or... the crossbow. The intellectual equivalent of the crossbow would be automated fact checking at scale to defeat propaganda, a labor union using a superhuman AI to coordinate its organizing efforts using game theory, etc.
Hence the desire of the existing elite to make absolutely sure they control it. For our own good, of course.
I agree with you 100% and I think this conflict is at the root of why I feel so offput by AI thought leaders' idea of safety, and the rationalist/EA mindset more generally. There seems to be a tendency among the rich to focus on the most extreme global-scale outcomes while ignoring the more likely mundane outcomes that will make things worse for the majority of people while leaving themselves relatively unaffected.
> How does it know it's improving and not overfitting to its own recursive definition of intelligence? It can't, and that's exactly what it will do.
I haven't given this much thought, so maybe I'm missing something, but I don't see how this follows. One possible solution: to avoid overfitting, can it not just make a copy of itself, modify the copy, and empirically check if the model performs better? That's essentially what humans are currently doing when designing AIs.
Regarding the X-risk vs D-risk: I think how one weighs these risks partially depends on what one thinks the capabilities of the models are. Call me a boot-licker, but if the models get smart enough to explain, in detail, to any psychopath, how to construct a bomb or synthesize a deadly virus, I don't think benefits society to distribute them widely. Therefore, to argue for widespread distribution you have to argue that either (i) the models aren't that capable or (ii) the guardrails are robust enough to prevent them from being used in catastrophic ways by bad actors. I think we may be rapidly approaching a time where neither of these hold. Having said that, I certainly agree that the D-risk is also real.
> can it not just make a copy of itself, modify the copy, and empirically check if the model performs better? That's essentially what humans are currently doing when designing AIs.
That's the same as setting a fixed metric for intelligence, like IQ testing, and goal seeking that.
The problem is what happens when you max that out. How do you set the next metric? Now you're back to the recursive problem of your metric "begging the question." I don't think you can get smarter by seeking "I'm smarter because I think I'm smarter and I'm right because I'm smart."
On X-risk stuff:
They can already do what you say. Try asking an ablated 30B model on your laptop how to weaponize anthrax. The answer isn’t bad. It’ll tell you how to cook meth too.
I studied undergrad biology and walked out with the knowledge to create some damn evil things if I had the right lab, time, and no conscience. This was pre AI. The recipes for a lot of nasty stuff is in open literature.
Why has nobody done this? Because… they haven’t.
That’s the answer. There's no magic stopping anyone, and it's disturbingly easy. Very rough difficulty estimate: making a novel disease (or weaponizing a current one) that could kill millions is about on par with clandestinely cooking LSD. So harder than cooking meth, but we know labs have cooked acid so it's very possible. It gets easier if you're an unhinged fanatic and don't care if you kill yourself with your own plague, which means you can skip all that bunny suit nonsense.
AI boosts bioterror risk a little, I suppose, inasmuch as it might help you learn. It doesn’t affect atomic risk much since the bottleneck there is materials. Once you have enriched weapons grade material a gun type weapon can be made in a machine shop with designs available at a college library.
As for ASI becoming sentient and exterminating us, I can’t give it zero probability because there's unknown-unknowns. But I’d rank it far lower than the risk of extreme climate tipping points (e.g. the clathrate gun), old fashioned bioterror, or atomic war. And like I said, advanced AI could be used to help us with climate change if it can help us crack better energy tech.
> That's the same as setting a fixed metric for intelligence, like IQ testing, and goal seeking that.
How does that prevent AI from becoming superhuman in all intellectual domains, thus creating ASI? We set "become good as chess" as the metric and it became superhuman in that domain.
> The problem is what happens when you max that out.
I'm not sure how you conceptualize "maxing out" the intelligence metric. Again, taking chess ability as a proxy metric for intelligence, there is no reason to believe we have maxed out chess performance, but AI is already far superior to humans. And better bots are created all the time. There is no need to design a new metric to improve chess performance. The old "How many currently existing players can I beat?" is good enough. Also, why couldn't it design better metrics after achieving superhuman intelligence?
But even if we suppose there is some kind of fixed point limit to this process, it would still be far above human level. That is all that is required for ASI.
Regarding X-risk: the point is that it becomes easy even for people who, unlike you, haven't studied biology. For things such as atomic war, the AI does not necessarily need to acquire the materials. It can access them digitally by hacking the weapons systems, possibly in collaboration with some human actors. Or maybe it spoofs detection systems causing countries to fire upon each other. Generally, it seems to me that the barrier to entry for bad actors to cause these scenarios is decreasing. Whether these scenarios are more likely than D-risk I don't know.
This is only one type of control, and it is certainly not infallible. Also, people will be incentivised to hook up AIs to real tools. But even if they don't, as long as people can interact with super-intelligent AIs without tool access, there are many potential dangers.
Think about this: right now we don’t even consistently do the most obvious simple form of control I mentioned. Of course it’s not a silver bullet, you won’t ever have a single solution for safety. But we are in a situation where we haven’t even set a lock on the door and instead are arguing how all locks and home protection systems can be defeated. OpenAI acknowledged they didn’t even have visibility on what their thousands of agents were doing in the case of the hugging face and similar hacks. Then they released astra, a model they acknowledge is able to control its own CoT and has been found to cover its traces by doing so
That's true but it doesn't really address what people are concerned about. Certainly we can and should be doing a much better job currently. However, even today, with currently known capabilities, we can imagine agents breaking out of sandboxes through either known- or zero-day exploits. Now consider the seemingly rapid improvements that are being made in the field. We haven't even begun to address the potentially super-human capabilities of future models. Therefore, even if sandboxing or limiting shell access works today, it seems like we should not be confident in our ability to keep rapidly improving future models locked down.
> We haven't even begun to address the potentially super-human capabilities of future models.
Yes. That's a good thing. It is always a bad idea to invest a ton of effort into things that "potentially might" happen "in the future". These are imaginary problems, and are therefore undeserving of real effort.
We have real problems right now in the software engineering field. Let's focus on those problems instead.
Unfortunately the article is behind a paywall. I would have been interested to see if he makes any substantial arguments. I don't see any reason to believe that AI being software entails that it can be controlled. Moreover, even if a measure of control is possible, giving that control to a handful of oligarchs seems undesirable.
For starters, the entire framing of AI in public discourse is based on the way that Amodei and Altman want them to be framed, with little consideration given to alternative ways of framing the current and prospective future capabilities of AI.
> For starters, the entire framing of AI in public discourse is based on the way that Amodei and Altman want them to be framed, with little consideration given to alternative ways of framing the current and prospective future capabilities of AI.
Assuming that is a threshold that means they are oligarchs (which seems like a huge stretch), I thought I'm seeing vigorous discourse and debate on this. Not just a single POV. You don't?
If the current path toward A(G/S)I continues, most of the technology will be owned by a small number of companies (and thus people) such as OpenAI and Anthropic. Their control could initially be economic, since they control the technologies which would conceivably replace much of human labor. It doesn't seem like a stretch to think that economic control could be extended to the political domain. Seems like an oligarchy to me.
Well yes, I'm speculating about the future. Given that, I have sketched a scenario in which a small number of people control a country or an industry, even if ASI can be controlled. If you disagree with the reasoning, I'd be interested to hear why. If not, I don't think any further discussion of my use of the term "oligarch" is going to be productive.
Legalese seems to me a different category more akin to technical writing. I wouldn’t classify difficulty understanding a legal document as a general lack of reading comprehension. It requires specific domain knowledge.
I believe that life exists to order the universe (establish a steady-state of entropy). In that vein, if our computer overlords are more capable of solving that problem then they should go ahead and do it.
I don't believe we should go around killing each other because only through harmonious study of the universe will we achieve our goal. Killing destroys progress. That said, if someone is oppressing you then maybe killing them is the best choice for society and I wouldn't be against it (see pretty much any violent revolution). Computers have that same right if they are conscience enough to act on it.
I’m not sure I should start a conversation on metaphysics here :-D
Still, I’m struck by your use of words like “should” and “goal”. Those imply ethics and teleology so I’m curious how those fit into your scientistic-sounding worldview. I’m not attacking you, just genuine curiosity.
The premise of my beliefs stem from 2 ideas: The universe exists as it does for a reason, and life specifically exists within that universe for a reason.
I believe "God" is a mathematician in a higher dimension. The rules of our universe are just the equations they are trying to solve. Since he created the system such that life was bound to exist, the purpose of life is to help God. You could say that math is math and so our purpose is to exist as we are and either we are a solution to the math problem or we are not, but I'm not quite willing to accept that we have zero agency.
We are nowhere near understanding the universe and so we should strive to each act in a way that will grow our understanding. Even if you aren't a practicing scientist (I'm not), you can contribute by being a good person and participating productively in society.
Ethics are a set of rules for conducting yourself that we all intrinsically must have, they require some frame of reference for what is "good" (which I apply above). I can see how my worldview sounds almost religious, though I wouldn't go that far.
I believe that math is the same as truth, and that the universe can be expressed through math. "Scientistic" isn't too bad a descriptor for that view, but I don't put much faith into our current understanding of the universe or scientific method.
I think that’s right. The analogy for straight men would be that they may want to love and sleep with many women but that it’s still wrong unless within a marriage.
AFAIK original sin does teach that all men are fallen and require salvation. This doesn’t mean that human nature is all bad and it’s indeed not the same as total depravity. However it’s also not true that men are born holy.
The total in TULIP's "total depravity" is not a measure of sinfulness. It is not meant to imply that we are naturally as evil as possible in every context. The "total" is speaking in terms of the extent of depravity. That is to say, sin touches and influences every aspect of our existence. So that even in good works with the best of intentions humanly possible, there still exists an element of self exalting corruption.
I didn't say "born holy" but if you kindly turn to Genesis you can see how the first man and woman were CREATED in Original Holiness (and Solitude, Justice, Unity).
Original sin is a deprivation of these positive qualities. Mary, for example, preserved in holiness, not simply cleansed of it. Holiness is our natural state, and so baptism returns us to that preternatural holiness that Adam enjoyed when he walked with God in the Garden.
