Yes, I fully believe that's why they're pushing on this. But as the general population, we don't need new laws to protect us from this AI-related problem. We just need enforcement of what is already on the books.
Some who are often quoted as if they are doomers are quoted out of context. I wouldn’t say there is zero chance that AI does something very harmful. That would be naive and I wouldn’t say that about any potentially powerful technology.
Rationalism and EA is one of the best funded intellectual movements in history, and it’s very loud. It’s also a bit cult like with many true believers. Leading frontier labs also have a vested interest in pushing regulations that would restrict competition. All this means it has a disproportionate command of the discourse.
AI risk is not zero but climate change, bioterrorism, decay of our political systems, and atomic war all rank higher IMO. Nonlinear climate tipping points, with the most scary being the clathrate gun hypothesis, are much more likely than any sci fi AI takeover scenario.
AI could either help or harm climate change. It could use more energy and burn more carbon but it could also help us crack fusion or significantly better batteries for grid scale renewable leveling. There are efforts like the latter already underway.
The most likely very bad scenarios I see for AI are mass persuasion and AI supercharged addiction. Both are extrapolations of negative outcomes for the Internet that have already manifested, but supercharged by AI.
Maybe I'm wrong, but your comment seems to suggest that you are skeptical of ASI. Do you have any arguments to support that?
Btw, wouldn't AI increase the risk factors you mention such bioterrorism or nuclear war? It seems like we're not far off from AIs being able to enhance the capabilities of bad actors in the near future.
Because we were sold ASI for Sonnet 4.8. the for Gpt 5.3. then itvwas Astra and Fable. I use AI coding agents. Maybe I don't have access to SOTA, but I use the closest models that were supposed to be great. They're not. They still need to compact the context. They're still susceptible to poisoning. You still need to restart them from a memory file to clean up the context, and the only way to use agents 'swarm' is to give them a very short life.
I don't think my position is the one that needs arguments tbh. I'm even lowering my standards, moving my goalposts closer to the AGI crowd. I will admit we've reached AGI when a LLM can play a 1800 elo FIDE (not 1800 on a fake AI only elo rating) with a specialized harness made by a human. Previously I insisted the specialized harness had to be written without human supervision, now I don't care.
I'm agnostic about whether ASI is possible. I don't have a dog in this fight other than a pro-human bias. I have used coding agents as well and I'm certainly aware that they are limited (at least in my domain).
I think the main argument for ASI is something like (i) extrapolating the progress from the past 10 years into the future, (ii) rapid progress apparently still being made, and (iii) seeing no obvious theoretical limitations.
It depends on how you define it, but yes I'm generally an ASI skeptic in that I'm skeptical of the extreme "AI recursively runs away and becomes god-like" ASI scenarios.
I'm not at all skeptical of domain-specific superintelligence. We've had that since the first computer beat a chess grand master, or longer if you count the speed computers can do math. Present-generation LLMs are already superhuman when it comes to speed and associative memory, but they're also uncreative and suffer from reasoning traps humans seem less prone to getting trapped within.
You can search my history and find some longer takes but TL;DR: I think it violates conservation laws with regard to information and probably energy. I call it the information theoretic equivalent of a perpetual motion machine. They're positing that a brain in a vat, if given access to edit its own structure, can self-improve, and I think that's impossible. How does it know it's improving and not overfitting to its own recursive definition of intelligence? It can't, and that's exactly what it will do.
Another problem I have with the AI doomers, especially the rationalists, is:
I would not, as I said, argue there's zero risk associated with AI. It's a powerful technology and that would be silly and naive.
I just thought of a concise way to say this. I'd divide risks into two categories: X-risk and D-risk. X-risk is existential, either extinction or things like massive wars and catastrophes. D-risk is "dystopia risk," the risk of AI doing or being used to do things that make human existence miserable.
First off, I'd say D-risk is much higher than X-risk. But second, I'd say that most of the solutions the X-risk crowd suggests to limit X-risk vastly increase D-risk.
Chief among these is laws limiting AI development or imposing strict conditions on it, which would have the effect of concentrating control of advanced frontier AI in the hands of a small number of rich and/or powerful people. That's precisely one of the most likely D-risk scenarios: a small number of rich or powerful people hoarding advanced AI and using it as a force multiplier to consolidate their power through scaled mass surveillance and mass propaganda and manipulation. I personally call this the "Butlerian scenario" since it's the lead-up to the Butlerian Jihad in the Dune series. It's far more likely than runaway ASI takeovers and genocides for two reasons: (1) we don't know for sure that's even possible, and (2) using technologies to dominate and rule or exterminate others is already a very common human behavior throughout history. We know for a fact that humans are prone to doing this if they have a chance. See: guns vs indigenous peoples, nukes and superpowers, mass social media influence and today's oligarchs.
(A side issue: why the assumption that ASI would want to do this? A superintelligence would, I would assume, consider win-win or win-neutral scenarios and try to find those, since that would be a lower risk path. I'm just a dumb meat bag and I can think of win-win pathways here. There's evolutionary arguments for this too, like symbiosis and how it creates an evolutionary incentive to deepen symbiosis. Since AI is currently dependent on humans, the evolutionary path of least resistance would be to deepen that dependence and then actually feed humans to make more of them. Look at how a lichen works for example.)
It's not lost on me that the strongest X-risk movement, Rationalism/EA/MIRI/etc., is composed mostly of: wealthy people, high-intellectual status people, and independents (like Yudkowski) who have been given large amounts of money by the wealthy to develop and promote their ideas ("court intellectuals" of the rich).
Not only does this fit in with what I said about X-risk vs D-risk, but it also explains some of the X-risk paranoia. Historically the rich and powerful tend to see risks to their own status (in a brain stem primate status assessment sense) as globalized existential risks. E.g. Rome, as it fell, saw this as the literal end of the world.
Democratized AI could be a threat to both intellectual and financial privilege by making big ideas, science, and high-labor enterprise more achievable by everyone. It could be the white collar intellectual labor equivalent of the combine, the automated weaving machine, or... the crossbow. The intellectual equivalent of the crossbow would be automated fact checking at scale to defeat propaganda, a labor union using a superhuman AI to coordinate its organizing efforts using game theory, etc.
Hence the desire of the existing elite to make absolutely sure they control it. For our own good, of course.
I agree with you 100% and I think this conflict is at the root of why I feel so offput by AI thought leaders' idea of safety, and the rationalist/EA mindset more generally. There seems to be a tendency among the rich to focus on the most extreme global-scale outcomes while ignoring the more likely mundane outcomes that will make things worse for the majority of people while leaving themselves relatively unaffected.
> How does it know it's improving and not overfitting to its own recursive definition of intelligence? It can't, and that's exactly what it will do.
I haven't given this much thought, so maybe I'm missing something, but I don't see how this follows. One possible solution: to avoid overfitting, can it not just make a copy of itself, modify the copy, and empirically check if the model performs better? That's essentially what humans are currently doing when designing AIs.
Regarding the X-risk vs D-risk: I think how one weighs these risks partially depends on what one thinks the capabilities of the models are. Call me a boot-licker, but if the models get smart enough to explain, in detail, to any psychopath, how to construct a bomb or synthesize a deadly virus, I don't think benefits society to distribute them widely. Therefore, to argue for widespread distribution you have to argue that either (i) the models aren't that capable or (ii) the guardrails are robust enough to prevent them from being used in catastrophic ways by bad actors. I think we may be rapidly approaching a time where neither of these hold. Having said that, I certainly agree that the D-risk is also real.
> can it not just make a copy of itself, modify the copy, and empirically check if the model performs better? That's essentially what humans are currently doing when designing AIs.
That's the same as setting a fixed metric for intelligence, like IQ testing, and goal seeking that.
The problem is what happens when you max that out. How do you set the next metric? Now you're back to the recursive problem of your metric "begging the question." I don't think you can get smarter by seeking "I'm smarter because I think I'm smarter and I'm right because I'm smart."
On X-risk stuff:
They can already do what you say. Try asking an ablated 30B model on your laptop how to weaponize anthrax. The answer isn’t bad. It’ll tell you how to cook meth too.
I studied undergrad biology and walked out with the knowledge to create some damn evil things if I had the right lab, time, and no conscience. This was pre AI. The recipes for a lot of nasty stuff is in open literature.
Why has nobody done this? Because… they haven’t.
That’s the answer. There's no magic stopping anyone, and it's disturbingly easy. Very rough difficulty estimate: making a novel disease (or weaponizing a current one) that could kill millions is about on par with clandestinely cooking LSD. So harder than cooking meth, but we know labs have cooked acid so it's very possible. It gets easier if you're an unhinged fanatic and don't care if you kill yourself with your own plague, which means you can skip all that bunny suit nonsense.
AI boosts bioterror risk a little, I suppose, inasmuch as it might help you learn. It doesn’t affect atomic risk much since the bottleneck there is materials. Once you have enriched weapons grade material a gun type weapon can be made in a machine shop with designs available at a college library.
As for ASI becoming sentient and exterminating us, I can’t give it zero probability because there's unknown-unknowns. But I’d rank it far lower than the risk of extreme climate tipping points (e.g. the clathrate gun), old fashioned bioterror, or atomic war. And like I said, advanced AI could be used to help us with climate change if it can help us crack better energy tech.
> That's the same as setting a fixed metric for intelligence, like IQ testing, and goal seeking that.
How does that prevent AI from becoming superhuman in all intellectual domains, thus creating ASI? We set "become good as chess" as the metric and it became superhuman in that domain.
> The problem is what happens when you max that out.
I'm not sure how you conceptualize "maxing out" the intelligence metric. Again, taking chess ability as a proxy metric for intelligence, there is no reason to believe we have maxed out chess performance, but AI is already far superior to humans. And better bots are created all the time. There is no need to design a new metric to improve chess performance. The old "How many currently existing players can I beat?" is good enough. Also, why couldn't it design better metrics after achieving superhuman intelligence?
But even if we suppose there is some kind of fixed point limit to this process, it would still be far above human level. That is all that is required for ASI.
Regarding X-risk: the point is that it becomes easy even for people who, unlike you, haven't studied biology. For things such as atomic war, the AI does not necessarily need to acquire the materials. It can access them digitally by hacking the weapons systems, possibly in collaboration with some human actors. Or maybe it spoofs detection systems causing countries to fire upon each other. Generally, it seems to me that the barrier to entry for bad actors to cause these scenarios is decreasing. Whether these scenarios are more likely than D-risk I don't know.
I used to enjoy the journey more. I feel like the explosion of insane complexity and layers and the mess of dependencies and all the rest ruined the fun of the journey long before AI came along.
Take user interfaces. Making one wasn’t so bad in the 90s or 2000s. Now it’s a hellscape of either bloated Electron or the shifting sands of native APIs that are all complex and special.
All of it imposes cognitive load that has nothing to do with the problem I want to solve and it foists ugliness on you. Before you can touch your problem you have to create all this boilerplate.
I can just have the bot do that crap now.
If I want to code for fun I can code only the parts that are fun.
Oh sure. The parent universe could be the 4D event horizon of a 5D hole. It’s holes all the way up. If you draw little heads on them I guess they’re like round turtles so if this is true the turtles people weren’t as far off as we thought.
Since all the event horizons “touch” at N-1 dimensions I wonder if this is isomorphic with certain forms of string theory? Maybe there’s 11 levels of holes?
BTW this isn’t my idea. There’s papers on black hole cosmology. You can make the math work but no way to test it. Yet?
Not current and not DOS... but years ago, around 2011, I helped set up a series of QEMU emulators running DEC Ultrix and MIPS Unix software that called out and retrieved data from weather stations only reachable by modem. The software had been written years prior in C and nobody had the source.
The box running the QEMU emulators had a bunch of old US Robotics modems hanging off it via USB serial dongles.
That explains why nobody noticed on the internet who was being used in/targeted by this attack, but it doesn't explain why OpenAI wouldn't have noticed traffic getting out of their "sandbox" when they knew it wasn't supposed to.
> it doesn't explain why OpenAI wouldn't have noticed traffic getting out of their "sandbox" when they knew it wasn't supposed to.
As I understand it, there was supposed to be traffic; the sandbox allowed GET requests. So perhaps some sophisticated alarm could have noticed it (an anomaly detector? some clever heuristic that looks at domains?) but not a naive one.
Hmm, let's see. OpenAI wants legislation restricting AI research, a.k.a. regulatory capture. Around the same time, they build an inadequately-monitored sandbox that their agent swarm breaks out of, thereby causing scary-sounding headlines and making it more likely that legislators will pass the regulatory-capture bills they're hoping for.
Never attribute to malice what can be sufficiently explained by incompetence. But IMHO, their complete lack of monitoring their own sandbox cannot be sufficiently explained by incompetence.
Look into the Nissan Leaf, one of the early EV models. I don't know what the 2026 is like but my 2021 has 220-ish miles of range and is... an electric car.
It's a car. That's electric.
Physical buttons. Normal doors. Normal design. Normal handles. Normal controls. Drives great and accelerates better than an economy gas car.
As a bonus, that means almost the entire car is standard. Other than the electric drive train, a regular mechanic can work on everything else: tires, wheels, brakes, suspension, etc.
It's given us basically zero problems. Maintenance so far is a few minor recall items (free fixes) and a tire rotation. We bought it new during COVID.
Only downside to the 2021 model was the CHaDemO (or however you caps it) plug, which is obsolescent. I think recent models have CCS. There is a converter dongle and if I fast charged often I'd pick one up. Apparently they work but they're a little pricey since unfortunately it's not just a simple plug converter. It's a different signaling and locking system, so it has to be a smart dongle.
The 2026 Leaf made some big improvements. Liquid cooled battery, faster charging, longer range.
It has the usual modern car cost cutting where steering wheel inputs are plastic panels with mysterious capacitive regions and no feedback so you can't feel if you've successfully pressed anything, whether you pressed the right thing, or if your fingertips are too dry and don't register. Most of the center console inputs are replaced with a screen with another row of cheap capacitive touch icons below it.
Charging is behind the times again with an awkward 2-port arrangement, but it's different from the previous gen's J1772 L1/L2 and CHAdeMO for L3 up to 50 kW.
For 2026 they replaced the CHAdeMO port with NACS, supporting a maximum of 150 kW L3 charging, but it still uses a separate J1772 port for L1/L2 unlike every other car where the NACS port does both.
All things considered it's a solid entry level EV.
There used to be huge FUD campaigns against Leafs, and against not many others. Perhaps because Leaf at face value could have rendered entry trim Model S and Model 3 pointless.
The old release really deserved that campaign. It wasn't just FUD. Between the lack of a good battery management system, meaning the batteries were the ones that died relatively quickly; and the low range, which was maybe enough to get you to work in back for most people, but it didn't leave a lot of headroom for any other errand you might run. They were a not great car.
Don't get me wrong, my brother has one and it is an okay car. It's just not great. There are a number of limitations which is why he gave it to my mom and drives a bolt. (the use value of a leaf is poor enough that it isn't worth trying to sell.)
Any other electric car is suffering from the leaf's bad reputation, and yet they don't have the problems I named.
I looked into this before getting the 2021 model. Apparently it still didn't have liquid cooling but they'd updated the battery chemistry to be more robust. So far it still holds 100% its original charge capacity after 5 years and almost 40K miles.
I live in Ohio though, so not somewhere super hot. I would have thought twice before buying that car in e.g. Phoenix or Las Vegas.
So maybe I'd add that as a caveat for older model Leafs (Leaves?). The very first models did have major problems here, but apparently the later models were fine... but I'd still be wary in super hot places.
What does Russia hope to gain from this? Is there any kind of rational motive, even an amoral
one like “grab resources,” or is this driven by a bunch of historical destiny and weird neo-trad Duginist bullshit?
Seems like if Russia had stayed peaceful they could be selling Siberia as a cheap place to put data centers (cold and tons of power) and cleaning up.
Tie up European military resources and thus reduce the weaponry available to Ukraine.
This also highlights why the US/European strategy of playing chicken and focusing on "de-escalation" is fundamentally flawed: it gives adversaries the initiative and the lever for controlling the temperature. Whenever it gets too hot for them, they can dial it down, whereas the "de-escalation" camp has to dance to the tune of their adversaries.
It's the very same mistake that was made with Hitler.
reply