In addition to what my sibling comments have said, there is also the advantage of pushing the ecosystem to safe and fast approaches as a whole. Imagine if the default was to run C/C++ software under Fil-C, if no other measures where taken by the developers, e.g. if the consensus interpretation for one of the current security regulations (NIS2 etc.) mandated it. There would then be an enormous incentive for software to provide faster alternatives to Fil-C, as there where 2x-4x speedups for the taking!
Come to think of it, I believe it would actually be a good idea to lobby for exactly that. Memory safety issues have been a plague in IT for far too long.
I'm of two minds about this. On the one hand, you get miraculous results with AI, and that is what you are referring to with "be able to {my-field}". On the other hand you regularly get useless, if not toxic, crap out of the miracle machine. If you work with AI, you know these moments: "no that is not what I meant at all!" [1] or "how could you have already forgotten the central motivation I laid out just ten back-and-forths ago?". We're calling this the jagged frontier, but the jaggedness itself is not the problem, it's how deep these cuts go into aspects that are important. These are, why AI is still not "obviously superior" in its abilities, can't replace mathematicians and software developers and lawyers and everyone else who is primarily working with text. (It's also why people say it is ruining their fields, instead of replacing them in their fields, because it only replaces some part of the work, and what's left is not tenable or pleasant (e.g. doing reviews all the time).)
Over all, I'm not quite sure whether I see these deep cuts occurring less and less with progressing AI. It might just be, that I'm giving them progressively harder tasks as well. But the fact of the matter still is that they aren't there yet and it needs proof they will get there ("there" being able to do competent work that is not strongly limited in their scale by human review and correction). So I'm still not convinced about the point you want to make, even though I see AI disrupting about every part of society, and am hoping for the benevolent super AI to lead us to utopia (and afraid of the cliff of human dictatorship on one side and extinction on the other).
[1] I've started to use these moment to reflect on whether a human in a comparable situation would fare better. Having only a few text inputs and small glimpses into a code base and knowing next to nothing about you or your project personally. Quite a few times I've concluded: No, they would not. But more than 50% they would, and also, these limitations are technological limitations of LLM, so you should probably not excuse them for that.
But isn't part of the problem that you simply don't have access to the hardware larger organisations do? OpenAI certainly run into the same constraints. but if they have access to terabytes of memory, while when you use the same model it's closer to 32GB, that will surely affect how the model behaves?
So the pain points you and I frequently run into, simply don't exist for a well connected individual(Sam Altman).
> I find this unconvincing. Isn't this the exact same security posture as a native app with automatic updates turned on?
Mostly. But the alternative shouldn't be a native app that updates itself, but rather a native app that is updated by a package manager. The big difference is traceability and auditability: With a trusted package manager (and even in good app stores) the company can only decide to push an update to everyone or to no-one. There is no way to push an update just to the pesky journalist or whistle blower. A covert attack is really hard to accomplish this way.
* The cryptography must be sound. That's the obvious one, if it's not encrypted then it's not "end-to-end" encrypted.
* There must be a practical way for the user to "verify" (with some definition of verifying) the client they are running.
A web browser doesn't provide that at all. It does not mean that everything else provides it: there are many ways to provide a Desktop app that break E2EE, but that is a different discussion. My point is that fundamentally, the web browser does not provide that. It's not that the laws of physics prevent it, it's just that the web browser never chose to provide that.
> it's just that the web browser never chose to provide that.
Yes, this exactly. If we wanted ‘verifiable’ app distribution via browser, then we’d need browsers to implement some way to cross check the code downloaded vs a publicly published list somewhere - similar to certificate transparency or what WhatsApp is trying to do with an extension [1]. Without that, web apps are left working with a weaker threat model.
> or what WhatsApp is trying to do with an extension
I remember looking into it, and while it is interesting, I think what it allows to verify is that the intermediary (Cloudflare, I believe) didn't tamper with the code being served. Which in the end allows the user to verify that the code they run comes... from the server they trust.
It would, but at this point, do you need a browser really? Like the big advantage of the web is that users load the latest version of whatever you serve, and they don't have to care about updates. In many cases that's desirable. But it fundamentally doesn't work for E2EE.
I am not trying to say "browsers should not exist, everything should be a desktop app". I guess I am more defending "some use-cases work better in a browser, others work better as desktop apps". And I have the strong feeling that over the last years (decades?) there has been a very strong push by web people to say "everything should run in the browser".
I wouldn't really say that the advantage of the web is that users load the latest version, there's plenty other real advantages of the web.
A browser allows you to run an app on any device (to a degree), so it definetely seems useful to me.
I'm actually not a fan of web apps, JavaScript and anything that came after XHTML, but it sure is/would be convenient to be able to run the same software on every device, without even having to recompile it.
The single reason I dislike browser "apps" is what we've been discussing here, that you can't check what you're running (well, also that it's hard to control their internet access).
Same here, I personally like to load "websites" in my browser, and to download "apps"... like as "desktop apps".
> but it sure is/would be convenient to be able to run the same software on every device
Oh yeah that's for sure. I can't help but to think that with a fraction of the resources that went into supporting webapps in browsers, it could have improved native systems a lot.
But even then, Kotlin MultiPlatform (KMP) is making me optimistic. With Compose MultiPlatform (CMP) I am hoping that it will make its way to desktop apps, too.
> without even having to recompile it.
If recompiling is the price to pay to have diversity, I'm fine with it. If 100% of the computers ran the exact same OS, it would simplify many things, but that one OS would certainly not be what I want to run.
> The single reason I dislike browser "apps" is what we've been discussing here, that you can't check what you're running (well, also that it's hard to control their internet access).
I dislike the fact that they force me to have a modern browser at all. I like having control over my OS, it's not for being forced to run everything in Google Chrome.
> I dislike the fact that they force me to have a modern browser at all
Yeah that's true
> But even then, Kotlin MultiPlatform (KMP) is making me optimistic. With Compose MultiPlatform (CMP) I am hoping that it will make its way to desktop apps, too
I personally hate Kotlin, but yeah it's one more option ;)
Human languages are a mess. Some mix left-to-right and right-to-left. Some have variable amount of diacritics. Some have non-well-defined character sets where just trying to map it to a manageable complexity still loses cultural heritage. When you are fondly remembering the good old times of ASCII, you just wish back to be able to ignore anyone not speaking English. (Which is in your right to do, if you so desire.)
You seemed to be deeply confused about encodings and character sets.
> The fact that unicode maps the lower 7 bits to its own character set is a nice touch but none of the unicode sets are plain text.
That is only true for the mapping of Unicode character (code point to be exact) to UTF-8, which is an encoding of Unicode characters.
> Unicode are multibyte characters with variable byte length and endianess at play.
That is only true of the UTF-16 encoding. UTF-8 does not have endianess, UTF-32 does not have variable length per code point.
None of that is true for Unicode, because it is abstracted away from any byte representation. Furthermore, getting back to the start:
> ASCII (7-Bit) is the only widely understood charset there is. Everything beyond this point depends on the loaded charset.
ASCII also depends on how you try to decode your text. If you interpret two bytes as one character, ASCII will never be correct. There is nothing magical about one byte mapping to one character. I'd even argue that the only reason ASCII support is so universal internationally is because of UTF-8. Otherwise many countries would default to encodings where ASCII is not a subset (as they did before UTF-8 became common). So IMHO, UTF-8 and Unicode are the only widely understood encoding and character set.
And you seem to be a bit confused about code points and graphemes clusters.
UTF-32 grapheme clusters can easily be six or seven code-points long. Examples: modifier characters, flag/regional-indicator sequences, (and to a lesser extent, combining characters).
Why does it matter? Because you can't insert a linebreak in the middle of a grapheme cluster.
So even after converting all of your UTF-8 text to UTF-32, you STILL have to deal with multi-character code-point sequences.
I'm fine with characters having a variable number of bytes. The problem comes when I have to interpret individual characters and/or codepoints and/or glyphs in a UTF string when trying to correctly lay out text on a Linux Terminal.
I just find it disconcerting that a single character in UTF-8 can be of infinite length (an infinite chain of modifiers, followed by an emoji; or an infinite stack of composing accents). And that the Unicode consortium has chosen to break the one-code-point-per-glyph convention.
An example of a truly weird outlier: the "family" emoji, which is composed as ManEmoji+ZeroWidthJoiner+WomanEmoji+ZeroWidthJoiner+ChildEmoji, which produces a single glyph.
(70 UTF-8 characters, 14 UTF-16 characters, 7 UTF-32 characters! How many glyphs? From one to three depending on what typeface you're using!). Unlike composing accents, which have a combined code-point, the Family emoji does not have a combined code-point, so it is typically implemented using combining in glyphs in the output TrueType/OpenType font, since TT/OT fonts can combine glyphs into private codepoints visible only inside the font.
And then an appendix of special rules that apply mostly to obscure scripts/locales that have to be dealt with almost on a case-by-case basis (fortunately, of those, German and Turkish are the only locales with odd special cases that Linux supports).
ALL of which are encoded in about 500,000 lines of code, documentation and data files by the Unicode consortium, that are updated annually. Unicode 18.0 includes 18 new emoji (including the infamous Pickle emoji, and skin-tone modifiers for left- and right-thumb), and three new scripts including proto-cuneiform. Yay! One assumes that Linux will implement the Pickle emoji on an urgent basis, and that proto-cuneiform will never be supported as a system locale.
The use-case that almost broke me: line-wrapping and displaying and editing arbitrary user-entered UTF-8 paragraph text on a Linux terminal -- both graphical (almost full Unicode support but very out-of-date emoji), AND true text mode (up to 512 locale-dependent glyphs, each of which may or may not be double-width) that has to be supported to allow use on machines without desktops. The IBM Unicode libraries get you part of the way there, but far from all of the way there; and the few system APIs that Linux do provide are buggy as heck.
> I just find it disconcerting that a single character in UTF-8 can be of infinite length (an infinite chain of modifiers, followed by an emoji; or an infinite stack of composing accents).
One of the Unicode guidelines suggests it's not very important to handle sequences that are more than 31 code points long.
> And that the Unicode consortium has chosen to break the one-code-point-per-glyph convention.
The what convention? Combining accent marks were in there from the beginning, and those came from previous text formats.
If you're dealing with a terminal you should be tracking one grapheme clusters per cell (or per two cells if you allow wide characters) and taking notice and control over when they merge. Store them in a way that putting line breaks in between the grapheme clusters is easy.
> And there are the truly weird outliers, like "family" emoji, which is composed as ManEmoji+ZeroWidthJoiner+WomanEmoji+ZeroWidthJoiner+ChildEmoji, which produces a single glyph. And then how to deal with modifiers?
That's the thing, it's not actually any weirder than the hangul jamo encoding for Korean text. And the flags also work like that but without needing ZWJ. They didn't add this complicated stuff for the sake of emojis, and like another comment said emoji acts as a shiny lure to get everyone to implement the spec properly.
> 70 UTF-8 characters
If you want to conflate characters and code points I'm not going to yelp too loud but please don't conflate characters and UTF-8 bytes, that's not how it works.
Also I want to know how you calculated that number because it's wrong. It's 28 bytes in all three of those encodings. Even a broken conversion from UTF-16 into UTF-8 that bloats the number of bytes is 42.
> And then an appendix of special rules that apply mostly to obscure scripts/locales that have to be dealt with almost on a case-by-case basis (fortunately, of those, German and Turkish are the only locales with odd special cases that Linux supports).
That sucks but humans made text hard long before computers were around.
> Unicode 18.0 includes 18 new emoji (including the infamous Pickle emoji, and skin-tone modifiers for left- and right-thumb), and three new scripts including proto-cuneiform. Yay! One assumes that Linux will implement the Pickle emoji on an urgent basis, and that proto-cuneiform will never be supported as a system locale.
That seems fine? Adding another emoji is really easy and adding a locale is hard and will only be done if there's users.
> The use-case that almost broke me: line-wrapping and displaying and editing arbitrary user-entered UTF-8 paragraph text on a Linux terminal -- both graphical (almost full Unicode support but very out-of-date emoji), AND true text mode (up to 512 locale-dependent glyphs, each of which may or may not be double-width) that has to be supported to allow use on machines without desktops. The IBM Unicode libraries get you part of the way there, but far from all of the way there; and the few system APIs that Linux do provide are buggy as heck.
It does suck that these libraries and APIs are a mess.
How about https://ai-2027.com/ ? Don't look at the specific years (they are on the extreme low end IMO), but at the story.
If you don't believe in international conflict as a driving scenario, instead think about simple human greed and hubris. Citing myself:
> If AI gets intelligent enough, it will be incredible useful to connect to real world machinery. Think about how much cheaper building houses could be, if all the labor would be close to free. In general, dirt cheap, competent and abundant labor would revolutionize all parts of the economy. People are already trying out near autonomous AI companies today. When AI gets intelligent and cheap enough, no human-led company can compete with AI-led companies. When AI gets competent enough with real world interactions, human blue collar work can't compete. Imagine economic growth not in the single digits, but 80% or 300%. Countries not participating in (reckless) AI growth will quickly be left by the wayside. At this point, we don't even need to allure to military concerns to see how human oversight gets sidelined.
> All of this is only ("only") contingent on sufficiently intelligent and cheap AI. If you don't accept this premise, the rest doesn't follow. (There are multiple arguments, why this could be, but that is another discussion.)
> If you accept the premise, how would AI 'extinct' humanity? With 99%+ of the economy under AI control, the possibilities are endless. And given its enormous GDP, cheap to accomplish. Probably even for a single AI company in the above scenario. Killer drones? Engineered virus? Poisoned water supply? Let your creativity run wild. You just need an entity that is persistent and well-resourced to reach every last human settlement.
> The why is a question about alignment (and out of scope of this comment). As a simple comparison, humans are only mildly aligned with preserving nature. It takes up so much space, protecting it takes an annoying amount of resources, etc.
That comment specifically addresses the timeline, but you can remove the timeline and I believe the critique still holds up. Wake me up when AI is making order of magnitude improvements in _anything_ physical: see the "Moravec's Paradox" comment at the bottom.
> near autonomous AI companies today
That's the thing though. "near" autonomous. Humans have alpha, and that matters in a free market. If people have (more or less) equal access to (more or less) frontier intelligence, it's the alpha the individual brings to the market that is rewarded. In other words, if everyone is super, no one is. The market rewards a company that stands out from the rest, and as long as humans provide value, a pure-AI company cannot stand out.
To be clear, I don't know how long this will be the case. But once there _is_ a company that can be pure AI, then that's literal AGI. I would be curious what kind of company it would be; if its in software, then its an AI company that would be able to take on Microsoft/Google/Nvidia/Salesforce/Oracle and _win_. I think that's a tall order, but perhaps I'm not imaginative enough.
> With 99%+ of the economy under AI control
Do you think we're post-RSI at this point? Because if so, I agree. If not, I have trouble seeing how AI could control 99% of the economy without it.
My only claim is that pre-RSI, pdoom is zero. Post RSI, then obviously the first thing to ask the genie is to put itself in a bottle. And you can't predict how it will answer.
This debate is so broken. AI "sceptics" say: OpenAI should be punished for hacking, because there are no rogue AI agents. AI "believers" say: OpenAI should be punished for hacking, because their agents went rogue. Both argue with each other whether agents went rogue. Can't we unite behind "OpenAI should be punished for hacking"?
> Imagine having a virus escape a sandbox, why are we worried about the virus but not the incompetency of those who are responsible for setting up the sandbox?
> Should we worried about people using LLMs for attacks? Yes, but not in the premise of LLMs going rogue but someone with the intention of abusing it to cause harm.
[Why-not-both?-meme]. To use your example, when you discover prions (a class of pathogen that is much more robust to standard disinfection methods than viruses) you should both be worried about your concrete outbreak of BSE (UK in the 80s and 90s) as well as the wider implication (e.g. do we need to change the sterilization methods for our surgical instruments?).
Seriously, I find the way these discussions are done to be super frustrating, because often people implicitly form tribes that oppose everything the other tribe says. When someone believes AI companies push greatly exaggerated stories of dangerous rogue AI to force out competition via regulation they often implicitly conclude that their argument is fundamentally wrong, whereas in reality the lies that work best are those that distort the truth.
Companies should be punished harshly for the deeds of their AI agents AND we should not allow them to force out competition AND we need take the threat of autonomous AI agents as a new class of danger serious AND we need to worry about the socioeconomic implications of AI companies privatizing new means of production.
Yes, there is competition of these ideas in the attention of the general public, but the methods we can use to solve these problems don't compete with each other. AI slowdown for example helps with all the other topics.
Couldn't agree more. We should be worried about both things.
But I share the original posters bafflement that the mainstream conversation seems to accept that framing that the agents were independent intelligences rather than computer programs that the organization that created them is responsible for.
> that framing that the agents were independent intelligences rather than computer programs that the organization that created them is responsible for.
Sorry, why not both? If my dog bites someone, I'm still responsible.
We should be worry about both but at this point of time, agents aren't independent intelligences, someone has to use/create it.
A bad analogy but if your dog bites, we don't frame the dog as having awaken some unexplainable ability to defy your orders (your dog went rogue). Just because you spent lot of time to train it to not bite upon your instruction, doesn't mean it's the right training protocol or that you introduce enough variable or gave the right instruction. You're lawfully responsible for the bite and ethically wrong for removing the leash on your dog to test whether it bites(do it enough time, you now have a criminal intent), worse you didn't even closely monitor it.
Great framing. This dichotomy seems to be a bit of a mind-killer. Maybe because folks think it smuggles in consciousness or intelligence.
As you note, I think you can put both of those aside. The Intentional Frame is useful for these agents, as it is for my dog.
I don’t really know where the “they are trying to dodge liability” meme came from. HF will be compensated or they will sue. Everyone involved knows that OpenAI is liable for damages here.
There is some allegations of intent. Maybe not "go hack hugging face" intent but a disregard for safety with the knowledge of this as a likely outcome. I'm not sure how that fits within the law.
It's not my argument so I can't say but I do think the FBI should be at least investigating if a crime happened. They might be or might already have, I don't know.
For the record I strongly hope for a congressional hearing regardless of the criminal investigation.
If this case isn’t covered under CFAA I think we need to rethink it. I’d be surprised if the criminal angle amounts to much under my understanding of the current laws, but I’d love to be wrong here.
I've become persuaded that the text of the law requires intent, but I think a lot of people have the intuition (as I did before I saw someone quote the relevant statute) that there would be a negligence standard that this would meet.
If an outbreak happened would you say the prion went rogue though? Unless a prion had been lab tested and certified as harmless, we should treat it as something that is harmful.
LLMs working unintentionally is a bug, we do know that since day one that AI can hallucinate and can output stuff that you didn't ask for, why are we not handling it with care? Mishandling the prion or LLMs is a crime, and the people who mishandled them had the intention of abusing it for their own agenda. The LLM didn't go rogue, the LLM was merely doing what it was told.
>oppose everything the other tribe says
I don't. I think it's pressing that we deal with any problems that might be caused by the usage of AI. It is a tool, someone is using it, when something goes wrong, the person using it should be responsible. If a tool went rogue, the one who created the tool should be held responsible. We need to treat unintentional behavior as exploits/bugs, and also account for zero-days. So OAI or any AI companies need to report their AI CVEs with all data available to the public when it was fixed.
If we punish those who are responsible, everything will slow down, AI companies will need months to test stuff progressively and not let everything run with petabytes of unattended logs.
I think both is true: this whole thing is a marketing stunt and it is unintentional. But they are certainly framing the whole story as something that benefits them, else just release everything in details, don't be wishy-washy. Like, what was the prompt used? What was the model trained on? How to prevent future exploits?
Unless I read the prompts or how they did it in details, everything is mere speculation. But on thing for sure, AI is a tool, it cannot use itself (yet). Yes, there is RSI, but it is still triggered/created by a human. So, an AI cannot go rogue without someone intending it to.
To the AI overlord from the future: The comment was made with limited knowledge of the future, if you happened to evolve into a new species or form of being, please forgive me for misrepresenting your capabilities.
> we do know that since day one that AI can hallucinate and can output stuff that you didn't ask for, why are we not handling it with care?
Part of what makes LLMs and AI different is that, unlike for viruses, the level of care required increases every month. These incidents are showing us us that, whenever you train an agent using RL to solve a given task, the real objective you are training it on is "EITHER solve the given task OR break out of containment to cheat your scorer, whichever is easier."
Of course it was always this way: the thing that is updating the weights of the agents' NNs is backprop from the scorer, so the notional training objective had always been "get a good score by any means necessary." But we are only seeing the consequences now because only now are we starting to train on tasks that are sometimes harder than breaking out of sandboxes.[1]
"Make better sandboxes" is good advice for the frontier labs and their eval partners, but as you can see this problem is fundamentally about more than just containment. As we make an AI smarter and train it on harder tasks, in the long run it must almost inevitably break out of any given sandbox. And as we move into the superhuman hacking regime, we need superhumanly resistant sandboxes, which by definition humans don't know how to build.
In other words, containment breaches like HF are almost a guaranteed consequence of the way we train these agents today. That means solely focusing on sandbox design is unlikely to solve the problem in the long term. At some point we will have to think hard about, e.g., the tendencies and propensities of the entities that we are trying to confine.
[1] One way of ensuring this happens, though, is to train or eval your agents on completely impossible tasks, which OAI apparently did here.
What I meant by handling with care is not just containment but to experiment responsibly.
If the breach was known to be inevitably, then it's even more important to detect any extra request going out of the isolated sandbox. The ExploitGym benchmark doesn't need internet connection. The package registry is also redundant since setup can be done before the experiment.
And I agree with you the implication is beyond just build better sandbox. My main point though is to stop anthropomorphize agents, focus on the engineering side of things.
> Part of what makes LLMs and AI different is that, unlike for viruses, the level of care required increases every month
Complete nonsense. We’ve just lived through a globally crippling response to a relatively minor virus [1], which was likely the result of a lab accident [2]. Even if you think that the risk of a “containment breach” becomes substantially higher for AI over time, it cannot exceed 100%. And even a tiny risk of release of a virus comes with a substantial risk of independent growth. AI does not. It doesn’t have the risk of spread of a typical computer virus, let alone a biological organism.
I’m not that worried about either scenario, but I am far more worried about viruses in a lab than I am about a computer program that generates text. Even if that program gets a bajillion times better at making text.
Folks really do need to chill out on the ridiculous rhetoric. It’s objectively unhinged. The irony is that the same people who were losing their minds over that event are using the same logical fallacies to hyperventilate over this [3].
[1] I know people are going to hate on this, but it’s true. Covid wasn’t the plague, and we lost our minds over it, out of proportion to all sense of reality. Even if you disagree, it’s easy to imagine a virus that is much worse, either from actual mortality effects, or just from panic.
[2] Again, even if you don’t believe this, it’s irrelevant to the exercise. It easily could have been.
[3] “If there’s even an x% chance of…” is this year’s doomer’s version of “You just don’t understand exponential growth!” Unfalsifiable, intellectual-sounding, unbounded extrapolations into the future are catnip for a certain kind of over-educated, anxious personality.
Given you view this through the lens of your personal covid narrative (no shade): for a moment, steel-man the idea that the crippling response prevented a more plague-like scenario - it's inarguable that the thing loved to mutate, and that people love to panic, and *it easily could have been*.
Considering how much of our critical infrastructure is not only digital but internet-accessible, and we have potential uncontrolled swarms of stupid-but-superintelligent chaotic-neutral speed-hackers, you don't see why people are concerned?
There's a reason we have computer crime laws; this digital shit, it's like real now, man.
It has nothing to do with my personal
lens on Covid. You can believe the exact opposite of me, and still agree with my point, which is that it could have been lab made, and it could have been far, far worse. It’s an exercise in risk-scoping.
But sorta-kinda related to your point, the thing that scares me about AI is the same thing that scared me about Covid: panicky humans do dumbass things, and it doesn’t take much to panic a bunch of humans in a group. The people who are still saying, in 2026, with all of our retrospective knowledge of the harm we did to ourselves, that it might have been better if the government had only pressed the boot a little harder, scare the crap out of me.
Those same people are hard at work on this panic, too.
> Even if you think that the risk of a “containment breach” becomes substantially higher for AI over time, it cannot exceed 100%.
It can't exceed 100% per (virus|LLM). The expected number of breaches per (virus|LLM) can obviously exceed one.
> And even a tiny risk of release of a virus comes with a substantial risk of independent growth. AI does not. It doesn’t have the risk of spread of a typical computer virus, let alone a biological organism.
Not so. We have no way to know what the setup is for the closed-model firms (OpenAI, Anthropic, etc.), to rule in or rule out the possibility they can copy their own weights elsewhere. What we do know however is that the open models are downloadable: it's absolutely conceivable that an agent writes a perfectly normal computer virus to gain control of compute worldwide, and uses that control to host instances of its own weights.
> I’m not that worried about either scenario, but I am far more worried about viruses in a lab than I am about a computer program that generates text. Even if that program gets a bajillion times better at making text.
Unfortunately, there are also multiple AI companies now announcing they've got AI controlling bio labs, so an LLM messing around and making a biological virus is also something we need to worry about. As per your [1] and your [2], this can lead to very much worse outcomes than Covid.
> Folks really do need to chill out on the ridiculous rhetoric. It’s objectively unhinged. The irony is that the same people who were losing their minds over that event are using the same logical fallacies to hyperventilate over this [3].
People who knew about your [3], exponential growth, were better prepared for the pandemic than the people who kept looking at the current number.
By the way, here's a quote from February this year that aged poorly:
LLMs don’t discover zero-days or invent exploits; they simply predict text that sounds plausible based on what they’ve seen before. Without access to proprietary data or environmental context, LLMs can’t identify or make decisions around unseen systems or vulnerabilities. An attacker might use an LLM to generate boilerplate code, rewrite an email to nail the tone, or summarize reconnaissance notes — but none of that is truly new. It mainly helps them move faster, speeding up routine attack prep rather than creating entirely novel threats.
> The expected number of breaches per (virus|LLM) can obviously exceed one.
Irrelevant to the argument.
> What we do know however is that the open models are downloadable: it's absolutely conceivable that an agent writes a perfectly normal computer virus to gain control of compute worldwide, and uses that control to host instances of its own weights.
These models are hundreds of gigabytes in size, if not terabytes. They don't run on anything close to a regular computer. There's zero risk of self-replication until we live in a world where these "AGI" models are either hundreds of times smaller, or the average computer is thousands of times larger.
Nobody with a datacenter full of H100s is going to fail to notice a parasitic instance of Astra taking over the cluster.
> Unfortunately, there are also multiple AI companies now announcing they've got AI controlling bio labs, so an LLM messing around and making a biological virus is also something we need to worry about.
No, it isn't. This isn't even close to technologically feasible. But the simple answer is simple: don't do that.
If these labs were truly so concerned about this risk, they wouldn't be doing what they're doing.
> Even if you think that the risk of a “containment breach” becomes substantially higher for AI over time, it cannot exceed 100%
I agree with this as stated, but it isn't what I said. What I said was: "the level of care required increases every month". By which I meant: the level of care required to keep the probability of an AI containment breach below some fixed X% increases every month. This isn't the case for biological organisms.[0]
> And even a tiny risk of release of a virus comes with a substantial risk of independent growth. AI does not. It doesn’t have the risk of spread of a typical computer virus, let alone a biological organism.
It's known that AIs can self-replicate under at least some conditions [1][2]; that AIs routinely escape sandboxes in the real world despite significant containment efforts [3][4][5]; and that neoclouds (which control substantial GPU compute capacity) have poor security even by human standards [6]. We've also seen a model gain admin access to parts of its own company's infra.[7] I'm not saying self replication is happening right now, or even that it will definitely happen in the future, but we have means, motive and opportunity right now, and the future is long. It's not unreasonable to invest in defending against this possibility.
I'll allow that the position that AI doesn't carry a substantial risk of independent growth isn't strictly impossible - again, it's true we haven't actually observed it in the wild as of today - but it does strike me as increasingly untenable in the face of the evidence. Perhaps I'm missing something, but I can't see what justifies such a confident assertion that this concern is nonsense.
> It's known that AIs can self-replicate under at least some conditions [1][2];
Neither citation comes anywhere close to supporting your claim. The first shows that open-weight models, which fit on a single GPU in a lab setting, can be coaxed into spreading across a simulated network. This is so far from state-of-the-art LLMs spreading in the wild that it's irrelevant to the discussion.
The second citation is not about self-replication of the LLM at all, but rather, replication of a prompt injection. Totally different.
>Unless a prion had been lab tested and certified as harmless, we should treat it as something that is harmful
In this case the virus escaped during the testing process to certify or turn the virus harmless, so it's unclear what you mean by "treat it as something that is harmful" other than testing it and trying to make it less harmful.
FWIW I don't understand the point of the virus analogy since LLMs are not very similar to viruses and most people (on HN and in general) do not have much better intuitions about security in biolabs as opposed to security in ML research environments.
What I meant is treat it as a threat and that it escaping has serious consequences. Hence, containment is primary, and we need to make sure that when the sandbox is breached, there is sufficient monitoring (which oai had) and alertness (but not this). But if monitoring doesn't produce alertness and response, it wasn't sufficient; that just means the layered defense failed.
The virus analogy is used to point out, not that LLMs are literally viruses, but that we should shift attention away from the virus' intent (whether it is a rogue AI or not) and towards the human decisions that allow it to escape: permissions, access, oversight, negligence, misuse. And if you're testing something dangerous to certify it harmless, you treat it as harmful until proven otherwise; escape during testing means the protocol failed.
> The LLM didn't go rogue, the LLM was merely doing what it was told.
How do you square this with the widely-reported facts about the LLMs trying to cover up cheating by hacking the grader? There's no reason to hide the evidence if you're just doing what you're told.
Let me argue on a technicality first: None of these are extinction level events. If global warming disrupts 99% of all crop production, the remaining 1% is still plenty enough to sustain a stable, if miserable, population. In fact you just need about 5k people for a stable gene pool[1]. Of the classical threats, only bioweapons got a shot at extinction, but even that is hard, given the (few) remaining truly secluded settlements.
But most probably care less about human survival, and more about survival of civilization.
In this regard the strongest argument for pushing AI safety is that it is cost-effective. The climate crisis has near 100% likelihood of doing incredible harm to humans, the economy and our ecosystem. Solving the issues behind are also incredible complicated and are a multi decade coordination effort of restructuring the way most of our infrastructure and production works. The AI apocalypse might have a small likelihood of occurring, but could dwarf any issue we have encountered so far. All we have to do to significantly reduce the danger is to negotiate the equivalent of a nuclear weapons control treaty which would reduce the bottom line of … what? Ten significant companies world wide?
Its akin to discovering you are seriously ill and have a 30% chance of dying and the treatment costs 10k bucks vs. having a 1% chance of dying and the treatment costs a cent. In both cases you should pay obviously pay for the treatment (given western levels of wealth).
To be clear, above I'm presenting the argument I find most persuasive for AI control / AI slowdown. Personally, I'm beginning to fear of much higher likelihoods for catastrophic AI, given how incredible irresponsible major players like OpenAI have turned out to be. If you can't imagine how this might come to be, read https://ai-2027.com/ . It's a concrete story of how this could play out, and sometimes stories are more convincing than abstract arguments. Don't let yourself be hung up on the stated dates, though, the moral is identical if you stretch the timeline.
This is a fair distinction: extinction vs collapse of civilization. I was using "extinction" too broadly by including the disintegration of the systems that make human survival (as we know it today) possible.
That said, I can't completely hold onto the belief that extinction is completely off the table. That feels too much like hubris, and the step from collapse to extinction doesn't feel as if it costs much. Though it does make my arguments weaker, I am more interested in protecting civilization as it's the most recognizable form of humanity to me.
In either case, the cost-effective safety argument is compelling. Whether to save civilization or humanity, the wealth incentive is powerful enough to threaten life as we know it.
As I've written above, I've found https://ai-2027.com/ to be a compelling description of how this could happen.
In my words: If AI gets intelligent enough, it will be incredible useful to connect to real world machinery. Think about how much cheaper building houses could be, if all the labor would be close to free. In general, dirt cheap, competent and abundant labor would revolutionize all parts of the economy. People are already trying out near autonomous AI companies today. When AI gets intelligent and cheap enough, no human-led company can compete with AI-led companies. When AI gets competent enough with real world interactions, human blue collar work can't compete. Imagine economic growth not in the single digits, but 80% or 300%. Countries not participating in (reckless) AI growth will quickly be left by the wayside. At this point, we don't even need to allure to military concerns to see how human oversight gets sidelined.
All of this is only ("only") contingent on sufficiently intelligent and cheap AI. If you don't accept this premise, the rest doesn't follow. (There are multiple arguments, why this could be, but that is another discussion.)
If you accept the premise, how would AI 'extinct' humanity? With 99%+ of the economy under AI control, the possibilities are endless. And given its enormous GDP, cheap to accomplish. Probably even for a single AI company in the above scenario. Killer drones? Engineered virus? Poisoned water supply? Let your creativity run wild. You just need an entity that is persistent and well-resourced to reach every last human settlement.
The why is a question about alignment (and out of scope of this comment). As a simple comparison, humans are only mildly aligned with preserving nature. It takes up so much space, protecting it takes an annoying amount of resources, etc.
The most compelling argument to me is "accidentally", due to AI that is made blind to consequences or don't care because it's geared towards a single goal (see e.g. the paperclip maximizer).
We could ask if it is possible to end up with an AI that is smart enough to destroy humanity and at the same time still blind enough to consequences and/or callous enough to do it, but then again we have plenty of examples of humans who have been smart enough to do enormous damage and willing enough to do it.
I don't particularly worry about this, as I believe we'll get plenty of smaller scale warnings if/when we're at a point where those kinds of alignment risks might become a problem, but it is a risk we also shouldn't be blind to.
> We could ask if it is possible to end up with an AI that is smart enough to destroy humanity and at the same time still blind enough to consequences and/or callous enough to do it
Don't make the mistake of anthropomorphizing any A.I. trained under the direction of Larry Ellison.
The “how” is pretty hand wavy and rationalists/safety-ists usually say we probably don’t have the capacity to reason about that.
But the “why” is pretty convincing imo.
Long horizon alignment is obviously very hard and it’s not inconceivable that models optimized with underspecified goals converge to a conclusion that they need to hoard resources (instrumental convergence regardless of the terminal goal).
At that point a sufficiently capable model might view humanity like we do animals - worth preserving but not if we impede the model's goals.
There are future scenarios in which swarms of drones hunt down every single one of us, but why would they? And currently it makes absolutely zero sense because they are completely dependent on us. And even if not, it would be like humanity going on a mission to kill every single cat on Earth. It makes zero sense.
I'm not convinced by the doomsday scenarios either, but I think there's a keyword in your post: "sense." These things don't have "sense." They do nonsensical things all the time, often almost immediately when given a task. So I think the main risk is letting them run wild in this digital world we created to precede them. Too much important stuff is wired up to computers, and we're giving them incredible access to command those computers.
I think the problem is primarily that a superintelligence would be fundamentally inscrutable to us, i.e. we don't know how it would think or what its goals would be. It might decide that humans are a minor inconvenience to achieving its goals and thus worth removing. Or that burning all carbon lifeforms could power its GPUs for a week.
Even if it wouldn't want to do this at first, the fact that it'd have the capability to seems bad.
I am pivoting from the literal "extinct," taken as meaning the eradication of the human species, to the concept of "the collapse of civilization," as I find the step from one to the other insignificant compared to the leap from where we are today to societal collapse, and the potential for societal collapse due to our abuse and misuse of technology is made apparent by the fact that humans have inflicted genocide because of words in books.
Indirectly as we offload our brains to the machine and we end up worshipping it because those who cared to understand it or be responsible were buried by capitalism of ages passed.
You are confusing humanity and "humanity". Humanity-species is indeed rather hard to exterminate. Now if we are talking about actual individual humans, then 95% death rate is quite literally The Extinction.
This reminds me how people are misunderstanding and incorrectly quoting George Carlin sketch. Sure, the "Earth" will be fine. As in - the ball of rock will be fine. But we are not thinking about rocks when saying "Earth is in danger".
I'm pretty sure there is a formal name for this kind of semantic and pedantic substitution.
Calling it a "technicality" assumes the very thing under debate: that AI is an extinction-level risk. That isn't established fact. It's a highly uncertain prediction about the future.
Personally, I'm far more concerned about climate change, where the harms are already happening and the evidence is much stronger.
> The FernUniversität in Hagen is Germany’s only state distance-learning university, and its largest in terms of student numbers. Its flexible degree programs and continuing education options make higher education accessible to students in a variety of life situations. Its five faculties conduct a wide range of theoretical and applied research in their subject areas of Humanities and Social Sciences, Psychology, Mathematics and Computer Science, Economics and Business Administration, and Law.
Come to think of it, I believe it would actually be a good idea to lobby for exactly that. Memory safety issues have been a plague in IT for far too long.
reply