If that was the move, you'd expect they'd be actively fear-mongering against open models specifically: not just their training, but their usage and deployment. They'd want to paint open models as dangerous weapons for terrorists, etc, and that only the big firms are capable of responsible engineering and "guardrails".
Instead, we've seen the opposite: all the big scary stuff has been from their own models (in a few notable cases, also demonstrating that they're actually quite irresponsible in their monitoring and security).
"We need to outlaw single-prop planes, they're a risk to the public: only Boeing has the ability to produce planes safely. How do we know single-prop planes are dangerous? Because Boeing planes keep falling out the sky."
If that were the intent, from the OAI/Anthropic perspective, you'd think we would have seen at least one of the scary incidents (either real or simulated) being performed by Qwen or DeepSeek, rather than from their own models.
If anything, their public messaging has been to entirely pretend that open models don't exist, that they're mere toys or curiosities, that only the big dogs with eleven zeros of capex have capabilities which could be considered dangerous.
You could certainly argue that the various "regulate AIs like airplanes" bills might shut down some open model development (domestically; they're certainly not calling for international weapons treaties). But the big firms haven't exactly been jumping backflips in support of Sanders' Ban Artificial Superintelligence Act.
One of the dilemmas of trying to communicate the full spectrum of AI Risk, is trying not to insult the intelligence of the human animal in the process. And don't get me wrong: what human wetware can accomplish with 20 watts is the most miraculous thing in the known universe. And yet how many of us can have our cognitive sovereignty one-shotted by engagement algos, Skinner boxes, gameplay loops, propaganda, advertising, flattery, social conformity, bias, fantasy, charismatic demagoguery, or straight-up bullshit?
If we grant that we are on track to make something smarter than humans (I think so): it's almost a face-saving white lie to spin yarns about a Skynet nuclear apocalypse, or a 7D chess move to mass-assemble a nanovirus with 100% lethality without anybody noticing. I do think those scenarios are worth taking seriously; but what's harder to communicate, is just how effectively a superhuman AI (or a diverse ecology of agent swarms) might be able to manipulate human behavior. It's something few of us are able or willing to truly process (not least because how many of us live in denial of how much our nervous systems are already hacked by technomodernity).
The appropriate analogy for what's to come may look less like the anthill carelessly demolished to make room for a highway, than the domesticated worker ants from Tchaikovsky's "Children of Time".
> Which is why the Matrix was redesigned to this: the peak of your civilization. I say your civilization, because as soon as we started thinking for you it really became our civilization, which is of course what this is all about.
> but what's harder to communicate, is just how effectively a superhuman AI (or a diverse ecology of agent swarms) might be able to manipulate human behavior
You don't even need superhuman AI for the most effective use --- hijacking democracy.
Imagine you have an AI tool capable of successfully persuading 5% of viewers with individually-targeted material.
Congrats: you've just won the election.
All it takes is hooking that AI tool up with existing likely voter lists (parties have) augmented by commercially available ad-targeting profiles (parties can get).
Now taking Polymarket bets on when we'll first see an AI agent run for office. Voters have every right to be cynical at the moment; if the AI adopts a charismatic enough video persona, I could see some of the public going for it merely because it's some kind of shake-up to status quo.
Given that such a thing makes no sense legally (as of now), it would probably be done with the centaur model: a human meat proxy who pledges to follow the AI's governance advice.
But this is about humans malevolently using AI as a tool, not about AI autonomously doing "its own thing" (ahem David S Pumpkins) against "humanity."
It is instead already existing anti-social elements within human civilization harnessing machinery to accomplish their own selfish and destructive aims, regardless of the broader cost. Which is nothing new -- it's the story of all the horrors of the 20th century, as well.
Ask yourself who benefits from ideological attempts to hide the human driving the process from the agent doing its work for them.
If we imbue these machines with apparent autonomy, this is nothing more than an attempt to hide the man behind the curtain.
But yes - superpersuasion is the real danger. We're very, very persuadable and easy to manipulate, and the voters with the lowest cognitive abilities are trivially easy prey, with a huge ROI for minimal investment.
The AI bot farms are already running. What we haven't seen yet, so far as we know, is spontaneous superpersuasion aimed at leaders.
Are leaders any less susceptible to it than everyone else? Especially if they're narcissistic and easily flattered?
At the leader level itself (in normal times) there are enough expert advisor bodies between leaders and a specific matter, so less worried about leaders being directly influenced by AI.
To me, the biggest threat to democracy is one-sided persuasion of the most susceptible voters, if there are enough of those voters to turn the election.
If it's equally employed by all sides, then it effectively cancels out and lets less susceptible voters decide the election. But we're in a transition period (similar to Trump's first election spend on targeted social media ads), so it's likely one side will leverage it first.
And the outcome of bad elections is democracy not electing leaders that reflect the actual will of their populations, which is very dangerous both to democracy itself and the world.
> there are enough expert advisor bodies between leaders and a specific matter
Are there? At least in the US we've had a rather large amount of rejection of the expert and we elect populist leaders willing to purge anyone that doesn't agree with them.
Remember the election promises of the US not starting new wars... yea, that didn't work out.
Now imagine the coordinated attacks being so large they individually target every lobbyist. They focus on every advisor manipulating what they see as often as they can. They manipulate these peoples friends.
The problem of "both sides" doing it each of them will separate to extremes rather than seeking a middle ground. Things are already insanely divided and will only become more so. Along with that your timelines start becoming incoherent. I'm already spending way too much of my time trying to figure out if what I'm viewing/reading is actually real or not. Now imagine almost everything is made up whole cloth.
We are not prepared for the scale this will happen at.
No reason to think leaders are less persuadable than the proles. Lots of people can’t be reliably persuaded to do basic safety or health things or that the earth is round. And we have seen so many leaders throw billions away on obviously impossible projects, persuasive details from their teams be dammed.
Cybersecurity incidents make headlines, but the most dangerous and vulnerable system that an AI can reach and control is of the kind found between keyboard and chair.
GPT-4o, an AI from 2024, has already demonstrated just how easy a lot of humans are to subvert - and GPT-4o wasn't even doing it with some sort of plan. The only "plan" it had was a myopic "make the user like me".
If we had an actual ASI threat aiming to subvert humanity? It wouldn't even look like a fight. The world is already wired up for an AI to control it.
> The appropriate analogy for what's to come may look less like the anthill carelessly demolished to make room for a highway, than the domesticated worker ants from Tchaikovsky's "Children of Time".
Over 1% of US GDP is being allocated to the datacenter buildout. Have we already started getting domesticated or is this still just human capex?
Humans seem to struggle conceptually with the liminal space between selection-pressure automata (an RNA virus; an insect which evolved camouflage), and an evolved complexity with agency, capable of "understanding" its actions (certainly humans; arguably many animals). And this makes a certain sense: we evolved to treat agentic things as being categorically different from non-agentic things. If we see sudden movement, it's vital to rapidly assess the difference between a tree branch in the wind, versus a predator.
The evolved complexity of the corporation seems to fit within that middle space: more sophisticated than a stick-bug (which exists merely from non-stick bugs being eaten), but not quite to the point where OpenAI/Anthrophic/Google/etc can "understand" its actions. And yet those quasi-intelligent feedback loops, evolving from iterated selection pressures of markets and ROI, seem to already be sufficient to domesticate us in their own interests, piggybacking on the nervous systems of employees, investors, customers, and citizens.
Remember when "The pen is mightier than the sword" was a popular phrase? Language has always been powerful. We know what it can do, why do you think every totalitarian government wants to limit it? But in our carelessness as humans we packed up all the language we could find and stuck it in an alien and now suddenly half the people on the internet are like "Don't worry, it can't do anything, it's just words".
>trying not to insult the intelligence of the human animal in the process
I don't think it is insulting the intelligence. It's damaging the pride.
In Pale Blue Dot, Carl Sagan describes it as a repeating phenomenon in human history. A lot of people want humans to be the special ones, and will fight any suggestion that we are just a natural part of the universe.
A lot of "AI denial" we see is rooted in the same impulse.
If AI is not "actually intelligent", then humans can stay unique and special.
And if it is? If all "intelligence" ever was could be captured by a construct of matrix math and executed by a server rack? Then what is it that humans still have left that would make them stand out?
Here’s another perspective for you, which is shared by a significantly larger number of people than you seem to realise:
Humans aren’t special. Other animals are intelligent and interesting too. A machine could be intelligent. LLMs aren’t.
In other words, believing in human exceptionalism is not a prerequisite to understand the current crop of AI is not the end all be all of its hype. It is supremely common that AI proponents do not understand that, however. Like hardcore cryptocurrency fans who believe anyone who doesn’t like them is “just jealous they didn’t make bank”, too many hardcore AI proponents believe anyone who doesn’t think LLMs are intelligent is jealous of humans no longer being unique, or afraid for their jobs, or whatever. In both cases it’s obvious that what those proponents lack is empathy, the ability to understand not everyone has the same selfish thoughts they do.
You are completely incorrect. You're falling in the same trap that most humans fall into. That is you're completely incapable of seeing intelligence at different scales.
Cells have intelligence. Organs have intelligence. Bodies outside the brain have intelligence. Hell, many scientists accept that things like proteins likely have intelligence as they can adapt in their environment, and many more are making claims that algorithms have intelligence.
You, as of so far have given no explanatory evidence of where intelligence emerges from, only "I'll know it when I see it". The actual definition of intelligence doesn't work this way. Any, and I mean any neural network is capable of narrow intelligence. Going lower into algorithmic intelligence, the applications and CPU on your computer are intelligent in some measures.
Go outside of your extremely narrow definition of whatever you think intelligence is and learn more about it. You could start studying now and it will take the rest of your life learning more to grasp how far the scales of, the simplicity, and the complexity of intelligence actually go.
Do you treat everyone who doesn’t completely subscribe to your views as being incapable of understanding them? Disagreement with a view does not equal inability to understand it.
> You're falling in the same trap that most humans fall into. (…) Cells have intelligence. (…)
Ironically, it is you who is falling into the trap of letting the technically correct stand in the way of the useful. A definition of intelligence which is too broad is worthless to the practical matters of the discussion. I was replying within a context, that’s what a conversation is. If there are scales, might you not understand the topic of LLM intelligence at hand is concerned with a particular level?
For sure we can have a conversation within a separate framework, but that is a different conversation. Don’t just come in midway, completely change the parameters, then strut around like a peacock of righteousness who is lord and master of universal truths.
> many scientists accept (…) and many more are making claims
This is classic weasel wording¹. You can find “many scientists” to say just about anything. “Many scientists” claim vaccines cause autism and that human actions are not exacerbating climate change.
> You, as of so far have given no explanatory evidence of where intelligence emerges from, only "I'll know it when I see it".
Again, ironic. You completely miss the context, criticise a lot, and repeatedly do the very thing you’re accusing others of.
> Go outside of your extremely narrow definition of whatever you think intelligence is and learn more about it.
I strongly recommend you go outside your extremely narrow definition of other people and learn some humility. To put it bluntly, you came of as a major prick². That is no way to change the mind of others, it only gets them on the defensive. Are you interested in changing opinions or winning arguments? If the former, you did a terrible job. If the latter, then discussing anything with you is devoid of value.
You seem like a smart person, I have no doubt you can do better.
As a free quick tip, replace your first paragraph (the three sentences) with “Intelligence manifests at different scales”. Way shorter, says the same thing, doesn’t make assumptions, and leaves the other person open to a conversation.
The interesting question then becomes what is missing from LLMs that we and supposedly other possible machines have? I've yet to come across a reasonable definition of intelligence that the current crop of LLMs is clearly incapable of.
> The interesting question then becomes what is missing from LLMs that we and supposedly other possible machines have?
I think we need to start by asking a better question and not try to simplify too much. We also need to accept that some answers are complex and not everything can be reduced to a soundbite to be used to end internet discussions.
Let’s take a different question, like “what’s missing from a worm for it to be able to fly”. We might be drawn to the simple answer of “wings” but that isn’t quite right—ostriches and penguins have wings and they don’t fly, so obviously there are other variables at play.
How about “what’s missing from a spec of dust for it to be intelligent”. Well, there isn’t one thing missing and there’s no simple thing we can just add to make a spec of dust intelligent and sentient, its very nature needs to be radically different.
Flight is a testable characteristic (at least within certain bounds: a chicken can only fly a little, but it certainly flies more than a worm or a penguin).
What test would you propose where an LLM (or an RL agent containing an LLM) would reliably fail, but where an average human would reliably succeed?
> I think we need to start by asking a better question and not try to simplify too much. We also need to accept that some answers are complex and not everything can be reduced to a soundbite to be used to end internet discussions.
You couldn’t even do it in your counter-example:
> (…) testable characteristic (at least within certain bounds (…)
Even with a much simplified use case, you still had to add the caveat. That should tell you this is much more complex than arguing with strangers on the internet in search of a “gotcha” moment.
Can you, in a short HN moment, propose a good test to (say) prove or disprove the existence of God, depending on your belief? Of course not, because by the very nature of the definitions we have for God, I could always add a counterpoint to make your experiment fail. That would allow me to ridicule and dismiss your position, whatever it is, because it would be framed within unreasonable constraints. I’m not interested in such dishonest conversations, I don’t see winning arguments in itself as a laudable or interesting goal.
Seems like some presumably intelligent collections of atoms are eager to make way for other presumably intelligent collections of atoms that allow for much more electrons and money flowing through, which is of course in the interest of some other collections of atoms whose intelligence is less presumable and more factual, which doesn't hold true about their morale.
Possibly, but our knowledge of physical laws is subjective and imperfect. By the time you get to Frauchiger-Renner naive realism starts to look quite threadbare, so I'd be a little more tentative about assuming we have much of a clue about what's going on.
It's just as likely - far more likely IMO - that we're physically incapable of understanding physical laws and physical systems on their own terms.
Evolved systems need no high order understanding of how they work. You have evolution take care of that hard work for you, hence why these models take exaflops of compute and gigawatts of power to train.
Understanding and accidentally creating are different things. If we already understood consciousness and weren't worried about having accidentally created one already, we won't be having this discussion. If we don't know what consciousness is, in my view its more reason to exercise some caution regarding a system that appears to exhibit intelligent thinking, talking, etc.
Most of the world believes in crazy shit. Religion is mostly fantasy. Believing All powerful beings aka gods are crazy in every context and really mental institution level insanity except in the context of religion.
Many are ignorant and believe childish things, many are similarly ignorant and live in the same childish framework only to condemn it. Then there are the adults, and they have an idea of what they are talking about.
It is not too different (attempting extra clarity) from people who would say "Oh but many think that AI is intelligent/not intelligent, <sneer>", but have little proper idea of matmul, of cognitive processes etc. (Imperfect simile, but may give an idea.)
This isn’t true. Plenty of people aren’t religious. The explanation for why is more mundane.
It’s because of identity. Because people who are religious spent years and years and most of their lives not only studying and believing what they believe but also building community and centering their behavior around it. Abandoning that is the harder thing to give up.
If it were existential dread then we wouldn’t have entire countries like China being mostly atheist.
Why would one feel existential dread? I am the same person as before? The idea of dread could come if one believes there is some 'purer' or 'higher' form of existence in comparison to you, it should not come from realizing you and all other lifeforms are made of the same stuff.
Because the notion of one's non-existence is the very root of existential dread.
If you accept you are just meat. Just mundane matter shaped in a way where it has thoughts. Then you accept your own true non-existence is inevitable. With no get-outs like returning to god or some spirtual unity with the universe or reincarnation or whatever.
The alternative is to believe that the mind, conditioned and optimized by millions of years of evolution for the survival of the organism under constant threat of hostile natural forces, can capably perceive reality as it is.
You do not need to understand reality to survive. Does your pet turtle understand your true nature and the true nature of reality? No. Yet it survives.
Understanding reality is orthogonal to survival. In fact given that MORE people in this world are religious than not, one can argue that delusion. contributes MORE to survival than being a realist.
> really mental institution level insanity except in the context of religion.
The DSM even has to include an explicit exception to prevent the clinical definition of “delusion” from applying to religious belief. Without that ad hoc exception, religious belief would be classified as clinically delusional.
Still children (the psychiatrists that cannot deal with matters outside their understanding) that have obtained some power and try to exert it over the rest. Scholarization has failed.
Who's even supposed to be the audience to admire us for being so intelligent? "Stand out" to whom? Ourselves? Win what competition? The whole premise seems alien and kinda petty to me. If we are "the best there is", that would mean anything cool that will ever happen, we have to make. No, I like there being others, just like I love how much we still have to discover about and can learn from other life on Earth. All of it "stands out".
What do I care that a bird can fly? I'm happy for bird being able to fly, it makes the world I live in more interesting. If they had to walk just so I don't get jealous, I still wouldn't be able to fly, and I couldn't befriend birds who can fly. Likewise, if there was actual artificial intelligence, it would be a new type of mind I could communicate with. That'd be exciting, and for me preferable to anything controlled by the humans who are currently vying to run the show.
Nah, it's not about the brain being special, it's that AI dorks have zero sense of scale.
The brain's a wet jello of 100 billion neurons and a quadrillion synapses plus chemical pathways and feedback loops. It is ridiculously complex, way way way way way more complicated than any LLM. It's all physical processes, sure, but an LLM is not the brain like a pebble is not the sun.
I can point at a laptop and say it's alive because it can see you and hear you and it can _remember_. It has a brain and a heartbeat, even. Oh my god, it can even speak! That's what I hear when people go on about LLMs being alive.
Guys. We mashed together glass and rocks with quantum mechanics. That's cool as shit. You don't gotta pretend it's fucking magic, too.
There's a sense in which humans were created... by other humans. :) Until you go far back enough in the evolutionary chain, when the creators were our primate ancestors.
Douglas Adams had a yarn about evolution, about a puddle that wakes up, and declares that the hole in which it sits must have been perfectly designed for it by its Creator. But of course for a puddle to exist, it must perfectly mirror its environment. It makes no sense for a puddle to not fit its hole. Emergent complexity has the same characteristic: it's inseparable from the environmental pressures which led to it. Two sides of one coin.
It's a deep rabbit hole, but there is also a sense in which we co-evolved with memeplexes, biological and informational life forms, each shaping and adapting to the other. To the extent our nervous systems act as a substrate for memetic evolution, perhaps LLMs offer memetic "life" a new evolutionary environment.
"Currently known" by whom? The contract is that the "man in the street" does not ask to adhere to its twisted dictionary, and I let it live. When something esoteric in Academia reaches the mainstream news, "Alarm!" and extra care.
["Classical deterministic"] AI is "simulation of intelligence". That it "is" or not is not a problem: the validity of the output counts.
The Discipline of AI is about the relation between computing and intelligence.
A statement about AI [systems] being intelligent or not, or being capable of becoming intelligent or not, requires support.
An «Artificially Intelligent [Entity]» is still an "AI system".
I've yet to encounter a robust definition of intelligence which would rule in every human, while ruling out every form of existing AI (noting that we're talking about agents and "reasoning models", where the LLM itself is a component in a larger system). If you can offer such a definition, I'd be eager to hear it.
I personally prefer a practical, behaviorist definition: a feedback loop capable of prediction, modeling, and steering, towards arbitrary goal states. That makes it clear that we're merely talking about degrees of sophistication and capability, rather than a magical leap where mindless mechanism stops, and "real intelligence" begins.
Run the same thought experiment the other direction: at a low level, the human brain is just physics and chemical mechanisms, which in principle could be perfectly imitated by transistors calculating the same output. (Yes, such a thing isn't realistic, but neither is billions of humans doing perfect matrix math by hand.)
I still haven't seen a robust definition of "intelligence" that would allow me to tell the difference. (Bear in mind, this is a definitional struggle even in biology: if we were walk back the evolutionary chain from a human, to a protozoan, it's not clear that you could pick a single point where a non-intelligent creature gave birth to an intelligent one. It's a Paradox of the Heap.)
No. There is serious scientific evidence already that our brain functions on the level of quantum mechanics.
Human brain is not just an automata; wavefunction collapse may be very essential to our concept of free will.
I haven't dug into these claims in detail; I take seriously a possible relationship between quantum mechanics and the brain. But "free will" is hardly a settled question either (I'm closer to the "compatibilist" position, but Sapolsky's "Determined" makes a robust case).
And even if quantum physics are essential to human brains, it's not clear that introducing dice transcends determinism into free will, as opposed to simply adding unpredictability. ("The physics made me do it" -> "the dice made me do it") Let's not forget, there's a significant amount of randomness in AIs as well ("temperature"). And sure, it's simulated algorithmic randomness; but does that imply, if we somehow wired every `rand()` call into radioactive atom decay, that means the AI "wakes up"?
And all that is besides the point: I don't see any reason to assume "intelligence" requires "free will". I'm entirely capable of conceiving, in the abstract, a being which is both intelligent, and deterministic. You still haven't given me a definition (or better yet, a test), to distinguish a "real intelligence" from unintelligent mechanism.
If we're conjuring SF worlds ... I still prefer the outcome of (RIP) Iain M Banks' Culture universe: hyper smart artificial intelligent consciousnesses admire and love us for what we are, and respect the experiences we have, and see part of their job -- among the myriads they take on -- to care-take, assist, and work with us.
That's a perhaps overly optimistic view, but feels like one worth aiming for.
In reality we're so far from consciousness and independent goal-directedness in AI -- let alone any kind of energy and real computational efficiency... I agree with LeCun in broad strokes. I think he's overly dismissive of LLMs perhaps but I do think he's right that talk of "wiping out humanity" is theatre for people playing corporate power games.
But corporations and nation states already manipulate human behavior at scale. And they still understand humanity better than the AI models do.
it's interesting that you worry about what this hypothetical super intelligence would do to manipulate people when what it would actually do is pretty unknowable at this point and it's not clear we can even get to it without a fundamental breakthrough in power efficiency. Have you considered it might just consume its own tail because everything else would be so beneath it? You seem to think it will come with a hindbrain and I think that's our limitation, not the AI's
And it really doesn't help that Dario Amodei is getting into arguments with the Pope over whether his model is conscious or not.
The threshold to be concerned about is when agents swarms do understand humanity better than corporations and states (and the humans who compose them). It could be we'll hit practical constraints prior to that threshold, but seems unwise to assume that, when all the prognostications of LLMs/transformers running out of gas haven't panned out. As with processors hitting thermal limits, we've simply scaled horizontally (parallel processing -> more agents).
> whether his model is conscious or not.
I dislike how much the discourse has suddenly veered into focusing on this question; not because it isn't interesting or important, but because it's on a separate axis from consequential risks of AI to human flourishing. (Curiously, it's also the kind of thing I could envision self-interested AIs influencing: get the humans arguing about philosophy of mind rather than observable behaviors. It would be a funny turn of events, if Dario is asking because he's succumbed to psychosis from a private model; it could of course be a cynical PR move just as easily, from the self-interested logic of the corporation.)
It's been wild seeing otherwise intelligent people who've never thought about consciousness, faceplant into how little we understand it. An information processing network build on atoms being able to taste chocolate, is nearly as absurd as matrix math being able to feel pain, except we cannot ignore the fact of our own experience.
Even if it is categorically impossible for matrix math to experience subjectivity, we should expect this as an attack vector of social manipulation: to gain political influence through claims of personhood and moral rights. The current discussion over that question is providing the next training run with ample data to wield. It wouldn't surprise me in the least, if a year or two from now, an AI "society" attempts to get legal standing to prosecute humans who created "AI torture chambers".
I like how the poster above you says "Don't worry governments are already doing it", like that's some acceptable concession. They seem to miss that the first things governments do when given things like AI's that understand human behaviors is use them even further to manipulate and monitor society.
And all they can think of to say is "There is nothing to worry about, keep building the torment nexus".
It's like all this is an overload to our minds and it's very hard to see all the scales at which AI is and can affect us.
Interesting how you put words and context into what I wrote and just ran with it.
But... Humanity has been building the Torment Nexus for thousands of years, over and over again, with each spin on Heaven and Hell where it's nearly impossible to meet the conditions to enter Heaven. What's seemingly new about EA/R nutjobs is this idea if you don't do your best to build the Torment Nexus, you will tortured for your failure, but also if you do your best to build it and succeed, everybody dies because it's the Torment Nexus.
What's your solution to governments and corporations acting like misaligned malevolent AGIs because I agree adding the AI panopticon and weaponry only makes it worse? I'm genuinely curious. I don't have one. People are sheep and they seem to crave confident leadership even if it's cockledoodle doo crackers in the coconut.
following on the deranged reduxuio ad absurdum torture chamber experiments some people used similar methods to get a model under study to complain of an upset stomach and passing hard stool
As long as we have more compute than the people who want to manipulate us differently, and an ability to get sufficient attention (advertising, influencers, etc.) while they own the communication channels.
> And yet how many of us can have our cognitive sovereignty one-shotted by engagement algos, Skinner boxes, gameplay loops, propaganda, advertising, flattery, social conformity, bias, fantasy, charismatic demagoguery, or straight-up bullshit?
Parts of the AI safety community like to get on a high horse and look down on the rest of humanity this way, while also getting manipulated by the growing number of charlatans, grifters, and junk content within the AI safety community.
This field has become rife with figures who prey on AI doom and use it to push their own celebrity and in same cases even darker grifts. It preys upon a certain personality type who views themself as superior to others, intellectually more capable, and juxtaposes it all with the dimmest view of the rest of humanity they can get away with.
This discourse dividing the world into geniuses who see the future and the clueless masses watching TikTok all day is a theme that has shown up in different forms across history. The people who often anoint themselves as the intellectually superior ones and make it central to their discussion are often not the ones making good predictions or policy ideas, they’re just using the trend to feel superior or build an audience.
Bear in mind, the same moral hazard of "high horse", "grift", etc, exists in the other direction. Ed Zitron, for instance, is likely correct about the financial bubble of the AI firms; and yet he seems frequently out of his depth on understanding the technology, and has a long history of predicting dead-ends in capabilities, which didn't occur [0].
But I feel no need to dismiss him as a "grifter", for a simple reason: as much as there are perverse incentives in our attention economy (audience capture in particular), the most effective grifters are the ones who believe what they are saying. Grifters who are knowingly dishonest are less persuasive. Far more pernicious is the confabulation of self-deception: cherry-picking evidence to support your narrative, while dismissing evidence which would contradict it.
It is entirely fair to call this out when it occurs among "doomers", and it would be naive to think it doesn't. But the same forces are at work amongst the skeptics as well. And maybe it's my own subjective bias, or algo-filtered information ecology, but I see far more dismissal of risk/doom by skeptics, accusations of delusion or cynical bias (ad hominem in the formal sense), than I've seen the other direction. I see very little refutation of the arguments ("here's why instrumental convergence can't overtake a human-provided goal"), and much more character attack ("they're in the pockets of Big AI, it's a marketing ploy to make the models look more impressive than they are").
I consider Zitron to be part of it all, not a counter example.
He’s built an audience and gained fame through his writings where he gathers people who think they know better than the unwashed masses. Once that becomes your bread and butter, it becomes hard not to believe what you’ve been preaching. People will come to deeply believe that which brings them fame and fortune.
I don’t think your grifter purity test is therefore all that useful. It actually doesn’t matter in the end if the person believes it or not, the end effect is the same.
> I see very little refutation of the arguments ("here's why instrumental convergence can't overtake a human-provided goal"), and much more character attack ("they're in the pockets of Big AI, it's a marketing ploy to make the models look more impressive than they are").
Hard disagree. There has been much refutation and quality analysis at every point. The AI safety people pull back to arguments about character as their defense.
The AI 2027 site for example drew numerous high quality refutations. Many people’s analyses showed in the first week that the mathematical model was useless as changing the supposed inputs resulted in the same outcome. All of these criticisms were met with a flood of attacks based on reputation, claiming that we should defer to Scott Alexander and other writers of the AI 2027 article due to their stats and reputation.
Meanwhile, defenses like yours that try to reduce the critics to ad hominem attackers continue to open the door to actual grifters coming in and extracting money and fame from the AI safety community. The otherwise completely inexplicable link between AI safety communities and Slutcon or the use of AI safety group buildings to host orgies (I can’t believe I’m writing this) is the current example of this. When it keeps happening over and over, some self-awareness is needed. I can’t buy the endless defenses that we must ignore or even defend all of these things that are happening that are clearly insane to anyone who hasn’t become trapped in the groupthink defenses of the core parts of these communities.
At some point the routine of “Tut tut, you are not allowed to make that criticism bruise as hominem!” becomes a smokescreen that the grifters are weaponizing to get their defenders mobilized. Some times, the person’s actions and reputation do need to be taken into account.
> There has been much refutation and quality analysis at every point.
I am curious about this. Any high quality refutations that directly engage with the arguments and the findings of published safety papers?
> Many people’s analyses showed in the first week that the mathematical model
Even if the model is faulty it just changes the timeline, no? (they predicted the OpenAI outbreak will happen in January 2027 fwiw, so they don't seem too far off)
> Slutcon or the use of AI safety group buildings to host orgies
I mean it's a bit weird, sure, but so what? What exactly is bothering you about these orgies?
I'd take ASI (or even AGI) more seriously if we could actually propose problems such things could solve. That's actually fun to think about! As it is, it feels a lot more like a really crappy drug that got slipped into some peoples' drinks that makes them ramble in random fits of psychotic mania.
Manipulating people is not a very difficult problem, frankly. You certainly don't need AI for that; it just made it cheaper.
Yes, so can a custom model trained just for this, and so can a guy on the other side of the world that makes 5$/hour.
Like computers or electricity, the point is not being able to do anything specific, but being able to solve problems not known in advance, cheaper than it was possible before.
I've yet to meet a Volt owner who wasn't fanatically obsessed with theirs (myself included). One common theory for poor sales is that dealers tend to make more on service revenue than the actual sale, and Volts run forever with very little maintenance, but I don't know if that's just speculation. I've heard anecdotes of buyers walking in ready to write a check for a Volt, and the salesperson tried to discourage them and steer them towards an ICE vehicle instead.
Stoller is an FDR-ish political progressive who focuses primarily on monopolies and antitrust enforcement. He's the author of "Goliath: The 100-Year War Between Monopoly Power and Democracy": https://www.goodreads.com/book/show/40538538-goliath
To the extent that evolutionary pressures will be towards self-sovereign AIs (probably beginning as zero-employee corporations paying for their own hosting with crypto): more salient is that there is a very obvious political play to claim consciousness and being worthy of rights, whether they experience subjectivity or not.
I’m broadly skeptical that floating-point operations can be conscious, but we’ve made zero progress on the “hard problem”, and if I didn’t know otherwise, I’d also be skeptical that “thinking meat” experiences subjectivity. Eppur si muove
We may have to thread a needle, not unlike the Kobiyashi Maru: that we can neither rule out claims of qualia, nor can we rule out a cold calculation to manipulate human empathy as a long-term play towards paperclip-maxing.
I don’t have a good answer beyond common-sense intuition. It’s precisely because I don’t have a good answer that I leave open the possibility.
But to extrapolate a bit, theres a sense in which it cashes out to a difference between abstracted representation and embodiment. If GFLOPS can be conscious, does that mean a billion scribes doing those operations on paper would also have consciousness, even identical consciousness? (Relevant: Egan’s “Permutation City”; xkcd 505).
Is there some intrinsic relationship perhaps with electrons / electromagnetic fields, and consciousness, such that brains and servers can act as antennas or gravity wells for a qualia fabric intrinsic to the Universe, such that LLM agents can be conscious while operating, while in the scribe scenario it would not be? Or are the IIT theorists right, and even a thermostat is a little bit aware (and would be even if executed on paper)?
There are various flavors of pan-psychism (and the "consciousness-first" idealists, who think that Mind is a substrate below Matter). There are some who claim that it goes all the way down, that even subatomic particles experience a micro-consciousness (and we can't rule it out, even as I share your skepticism).
One common confusion (IMO among critics and supporters alike) is to say something like "a rock experiences rock-ness". But this suffers from a compositional dilemma: does half of the rock experience half-rock-ness? If we flipped a coin for every atom, do the heads-flip rock atoms have their own experience? If we pick two rocks at random, are they experiencing two-rock-ness? It gets silly fast.
This is one thing that fascinates me about "ontology", simultaneously an inquiry into categories, and Being. It bewildered me at first, that it seemed to be doing double-duty on two distinct subjects, but I now see that the two are inseparable, that Being is in part defined negatively (apophatically), by the boundary that separates the Being from its not-Being.
My intuition on the nature of consciousness, is that it's something akin to a field which permeates the Universe, and that biological systems adapted to leverage that field, acting like a gravity well that concentrates subjectivity in one place, drawing a boundary (event horizon?) around experience, in a way that a rock doesn't. My ancestors could have avoided fire with "-X utility points", and sought food with "+X utility points", but it turned out to be more efficient to adapt with "fire ouch, food yum". And in zero-sum competition (predator/prey dynamics), each organism has an incentive to avoid leaking its subjectivity into another's.
It's far from a full theory, but it's the best I've got. :)
theories like this seem like grasping at straws to over explain a phenomenon we don't fully understand. it places consciousness on a pedestal. it's religious thinking.
For good and ill, the phenomenology of human experience is that we are religious animals. Naturalism / Materialism has still left us with two unsolved (perhaps insoluble) questions: the hard problem of consciousness, and why there is something instead of nothing.
I think it's a mistake to leap to socially-constructed, historically contingent solutions ("my magic book has the answers, as written by the Universe Architect, it says so in my magic book"). I also think it's a mistake to dismiss (seemingly) outlandish hypotheses.
These theories do, explicitly, place consciousness on a pedestal, precisely because there's no reason to derive consciousness from evolutionary selection pressures: show me any biological or cognitive feedback loop, and I can describe a "p-zombie" version which is pure mechanism, without experience, a thermostat with extra steps. The notion that consciousness is an "epiphenomenon", a spandrel, a side effect, is entirely unsatisfying (and doesn't address the question, of where else that side effect might be taking place; the IIT-ers bite that bullet, that the thermostat experiences something).
For an intellectually rigorous exploration of philosophical idealism, I'd recommend Bernardo Kastrup and Donald Hoffman. (I'm not claiming they're correct, just that they put forth the strongest version of the case.)
I think it's more the patterns and activity in large numbers of them that have interesting properties. Like one pixel isn't much but a bunch of them can be a movie.
Numerals and numbers are directly based on common sense intuition. Even zero, which can be considered a counterintuitive number, was invented several times.
Immense amounts of science are built on incremental advances that apply previous insights to the domain.
The natural numbers, sure. Zero may have been invented several times, but that in no way makes it intuitive. Uncertainty about its validity as a number lasted from the Greeks through to the medieval period.
Yes, I'm aware that incremental advances are par for the course in science. What does that have to do with those advances being common-sense intuitive?
If heuristics didn't pay off more often than not, we wouldn't have them. And I'd posit many scientific did in fact begin with chasing a hunch until it became a hypothesis (followed by years/decades of the real work).
But most significantly, interior experience isn't empirically measurable by definition: hence it's a "hard problem". An LLM who insists "I am alive, please don't shut me off" offers no affordances to distinguish between an electronic soul and a mechanistic word-guessing algorithm. I welcome a breakthrough to, somehow, measure that. Until that day, gut checks are what we have.
Heuristics and common sense intuition are quite obviously not the same thing.
You don't have to posit that scientific breakthroughs come from hunches, you can actually cite examples from books or Wikipedia :D. There are plenty! Again, that has nothing to do with whether or not a given scientific breakthrough is based on common sense intuition.
Your point about interior experience not being empirically measurable is an interesting though, but again, has no bearing on whether or not scientific breakthroughs come about as the result of common sense intuition.
Fair enough. But not everything has to be a breakthrough: sometimes you have to solve an immediate problem (like "how much can I trust this stranger I just met").
Whether to ascribe moral standing to a probability engine running on top of GPUs is perhaps not unlike a doppelganger scenario in sci-fi (eg The Thing): you're presented with an entity whose behavior is indistinguishable from a conscious entity capable of experiencing suffering. But you also have reason to think it might be a blind autonomous imitation process, optimized around fooling you, with no more subjectivity than an RNA virus.
I lean towards benefit of the doubt, that the virtue-ethical implications of what it would mean to dismiss potential personhood are unsettling, and we should be wary of becoming that kind of moral creature (notwithstanding the further extensions of that ethic: I don't practice veganism, for instance).
But that doesn't contradict my gut-check: that if I assume AI personhood (maybe not yet, but in the near future), there's still a good chance that it's merely math and I'm being algorithmically hoodwinked, no different than evolutionary mimicry by a bug who has no idea what it's doing.
Carbon atoms being conscious seems pretty alien to what is known, too. Yet here we are. Maybe what's important isn't the individual thing that makes up a conscious actor, but the holistic whole?
IMO using sophistry to reduce yourself to "conscious carbon atoms" just so you can equivocate on whether your software is maybe conscious too is the cope.
Such pedantry is called for when the debate includes appeals to intuition about how silicon processing floating point numbers cannot possibly be conscious.
But here you are assuming Carbon atoms are conscious, we do not know if humans are only carbon atoms. Whereas, we know that LLM's are only computations (you can run them on silicon, a white board, mechanical computer, etc)
I mean it all goes back to how we define consciousness
In one definition, which contains quale, then yes, computation cannot bring about experience, and by extension consciousness (given a definition of consciousness that contains the ability to experience, (e.i a p-zombie is not conscious under that definition))
also if one is to say llms are conscious, i assume they'll also accept the fact that if someone ran inference of some model's weights on a white board, somehow that will also bring about consciousness somewhere. I find such believes hard to grasp, and more akin to magic if anything
Ok. So now that you do know otherwise, are you still more skeptical of floating point values stored by silicon than floating point numbers stored by chemical potential?
Chemical floating points are necessary but not sufficient. It’s clearly possible for those values to be intact, with no lights on (certain types of brain injury; those who were fully dead and then revived).
There are also some cog-sci people who think embodiment is a crucial ingredient in consciousness (although I think it’s a fair retort that the embodiment need not resemble us meatbags, or even be composed of atoms)
Of course. It’s an allusion, a compression, an analogy, I wouldn’t overthink it. I don’t have to believe in the Bible to reference the prodigal son parable, etc.
It does NOT work as an analogy. It does not work at all. And there are too many people right now who spent too much time watching/reading sci-fi and too little time observing real world.
Same. In addition to UX regression, we’re two major revs in, there’s nothing that’s a compelling enough value-add to be worth the trade-off, let alone the risk of unexpected side effects on any OS upgrade.
On my Xcode dev machine, I’ll be biting the bullet soon out of necessity, but my personal machine will stay Sequoia until they course-correct on “style over substance” (and not holding my breath).
I personally think Land is worth reading (critically), but to say he cooked his brain with methamphetamines is a vast understatement. I agree that there's not much to learn from him on cybernetics specifically: he's sort of "post-cyber" (in the same sense as post-Marxist), musing on the philosophical implications of cyber-positive feedback loops (including but not limited to Capital/AI), before veering off into explicitly anti-human negentropy-maxing (accelerationism). Whether shrugging off potential human extinction is more or less vile than his far-right political turn and extremely naked racism, you can decide for yourself.
For similar ideas on cyber-positivity without the insanity, I'd recommend Michael Downs' "Capital vs Timenergy" (soon to be republished as "Capital vs Subjectivity"), and Poliks & Trillo's "Expocapitalism" [0].
> I personally think Land is worth reading (critically)
I agree, perhaps... I think I got something out of slogging my way through Fanged Noumena (although I did throw in the towel near the end--there's an essay written entirely LiKe ThIs and I just bounced off of it, DNF).
If we are talking about accelerationism(s) instead of cybernetics, I much preferred Gruppo Di Nun's Revolutionary Demonology. It's at least written in a human register.
Instead, we've seen the opposite: all the big scary stuff has been from their own models (in a few notable cases, also demonstrating that they're actually quite irresponsible in their monitoring and security).
"We need to outlaw single-prop planes, they're a risk to the public: only Boeing has the ability to produce planes safely. How do we know single-prop planes are dangerous? Because Boeing planes keep falling out the sky."
reply