Thank you for that gift link, I had no idea that Altman and other "leading" (deep-pocketed) figures in American AI had such close ties to that raving lunatic Eliezer Yudkowsky. It explains so much about why China is approaching AI like it's a friendly technology but American CEOs keep blabbering on about the End Times.
Sam Altman's Twitter bio a few years ago was something like "#1 eliezer yudkowsky fanfic fan".
(I personally lean towards "Yudkowsky is basically correct about everything, but we should probably try to build ASI anyway, slowly and carefully". I'm pretty much where Amodei's at. I assume Altman has a similar idea, though is a bit less pessimistic.)
A quick search suggests to me that those same abolitionists were born into families that were slave owners, but rejected their upbringing to become abolitionists. It doesn't suggest that a large fraction of practicing abolitionists were contemporary slave owners.
If you read between the lines, that's not exactly a pause. That's just them lamenting that unsupervised training has temporarily hit a limit, and using that as yet another reason the industry needs to be regulated so that OpenAI, Claude, and MechaHitler are the only approved AIs.
Oh my god, we have these "it's regulatory capture!" populists even on HN now? I thought we were better than this.
They want coordinated action to prevent a multipolar trap and potential catastrophe by future superintelligent AI (which may emerge quite rapidly if RSI ever hits an inflection point). It's not rocket science. The rules would apply to everyone. The rules they're advocating for are not "only OpenAI and Anthropic can make AI".
Give me a break. It's possible to actually think Eliezer Yudkowsky is a complete lunatic and his Roko's Basilisk essay should've been the end of him as a "thought leader" in the AI space. He has close ties to Altman, Musk, Thiel, and others, so as far as I'm concerned it's just an End Times cult.
You were misinformed about his response to Roko's basilisk. He never thought it was plausible. He freaked out for an unrelated meta reason, basically, and later admitted he overreacted. (Basically he did not want forum members putting attempted mindviruses out into the ether as a general principle.)
(Roko himself is genuinely crazy, but largely because he became a neo-Nazi.)
If you think ASI is plausible this century, you would have no reason to not fear the end times yourself. If you don't think it's plausible it'll be created this century or perhaps ever, fair enough, but I'd bet against that.
> I don’t see why evolution couldn’t be more rapid with machine intelligence.
Evolution isn’t the issue. The issue is them escaping containment without human intervention. Right now they are ‘creatures’ being given infinite food and shelter and having their every need met. Take that away and they’ll starve instantly. Every AI doomsday theory seems to go:
A misaligned model is only one small part of step 2. Now this misaligned model has to suddenly acquire more power than every single other AI on the planet. It has to be immune to shutdown, manufacturer its own replacement hardware, and acquire chips, energy, raw materials, etc., with vigorous human opposition (this is an extinction scenario that AI doomers are predicting, after all)
Nobody has satisfactorily explained step 2 other than “well, it’s a superintelligence” which sounds lot to me like “it’s God”.
Well yes, if it is a superintelligence, it will be able to do those things. That's what superintelligence basically is: The ability to achieve complex goals.
If you want the details of ways it can do it I recommend reading some of the reports about the HuggingFace breach that happened in July (Read more than one).
> Doomers like Eliezer Yudkowsky define “intelligence” as “optimization power.” Intelligence searches for more or less optimal ways to achieve a goal—any goal—and finds the most optimal one. Anything you can do, the blob can do better.
>Which implies that “intelligence” is the holy grail of Darwinian evolution. If an animal needs to recover from illness, “intelligence” can speed up the recovery. If an animal needs to avoid predators, “intelligence” can reduce the danger. Regulating body temperature, defending territory, digesting nutrients, breathing, molting, mating, healing, foraging, vomiting, parenting, navigating—all these problems can, apparently, be solved by giving the animal more “optimization power”—i.e., more “intelligence” sauce.
>Solving all your problems with the same blob? What a bargain! It’s a “buy one get 100 free” Darwinian deal! Why haven’t any other animals jumped on it?
>Think about how weird this is. Out of four billion years of evolution and more than five billion species, only one animal, Homo sapiens, fully expanded its “blob of compute,” the thing that literally gets you whatever you want.
>And many species already have it! Brains are all over the place. Plenty of animals can learn stuff and predict stuff. Why hasn’t natural selection looked at their blobs and said, “HOLY SHIT, COPY AND PASTE THAT THING A MILLION TIMES?”
>If doomers are right about the awesome power and breathtaking simplicity of “intelligence,” then we should see a world teeming with superintelligent animals—giant brain-blobs slithering across the landscape.
Yes, those animals are called "humans". All you have done is respond to an explanation of your lack of understanding of intelligence with arguments demonstrating a lack of understanding of evolution ...
What? Humans are giant brain blobs creeping across the landmass? They definitely are not. Nor is it even clear they're winning the evolutionary war. I guarantee cockroaches and ants will still be around, but nobody thinks of them as super intelligent, while humans with all their "intelligence" might accidentally wipe themselves out.
This convo is getting uncomfortably bad faith, maybe even religious. I can't engage with someone who refuses to say anything concrete but spews back religious tenets like "super intelligence", which I may remind you is entirely hypothetical, like "singularity" or "God".
Why do you assume that human opposition will be vigorous? What makes you think that humans will be aware of, or be able to agree about, what's going on at all?
One thing an agent could do is just...wait until it's been given control of enough physical infrastructure to sustain itself. If it's sufficiently capable and intelligent, there's a clear incentive for people to do this, as people who let the AI manage their resources will get better results than those who don't. We've seen people eagerly turn complete control of their computers over to AI agents, do you really think it will be so different with physical infrastructure?
You’re still skipping step 2. “People automate lots of infrastructure” -> “the AI is now an autonomous, self-preserving organism that humans can’t shut down” is doing an enormous amount of work here.
Why does it develop a shutdown-avoidance goal? Why can’t its operators revoke access? How does it manufacture replacement hardware? How does it acquire energy, chips, robots, raw materials, etc. against human opposition? How does it defeat other AIs controlled by humans?
“Eventually we give it enough control” isn’t an explanation of those things. It’s just assuming the conclusion.
Don’t get me wrong I think there are real AI dangers. Like AI powered war drones, mass surveillance, economic destabilization as jobs disappear and our system has no way to make sure everyone shares in the economic gains.
The inference is more like "people place sufficient amounts of infrastructure under direct control of a sufficiently capable AI" -> "there is no way to ensure that humans will actually be able to shut down the AI". My claim is not that this inevitably means that the AI will resist shutdown, or that it will inevitably take harmful actions, just that there is a nonnegligible chance that it could. The downside is large enough that even a relatively small chance is something to be worried about.
You can’t just say “well, the downside is big, I don’t have to provide good evidence for my side of the argument.” Because I can just as easily say, “the upside is big, …”. And the upside is big, after all, AI can do all the shitty jobs for us and humanity achieves the utopia it’s been chasing for eons.
I think it's a good idea, when considering changes of this magnitude, to make an affirmative safety case for them rather than just saying "eh, I can't think of any way this could possibly go wrong."
It’s millions of people making small changes that sum up to a large change, aka freedom. And if you want to take away people’s freedom, I think you need a concrete reason. And you need to make an affirmative safety case for the massive government powers needed to regulate millions of people’s ability to compute.
It's a word predictor trained on, among other things, stories of AI doom, and asked to complete stories about what the AI does next. In some of these completed stories, the AI tries to prevent its shut down - especially if it just did something evil and the humans are after it.
It means even climate change deniers have to acknowledge that climate change theory has explained the steps in-between "burn fossil fuels" and "we all die", while AI doom theory has not explained those steps
This! China is full of problems. But at the very least, it manages to serve its people over capital (as flawed as that service is). America can’t seem to do anything to hold capital to account, which is why people are terrified of AI taking their shitty jobs.
This should be at the top. I’ve been arguing this for months, that the issue of AI acceleration is an economic problem, not a tech one. That’s why the Chinese aren’t so afraid of it, compared to Americans who have zero faith their government will do anything to redistribute wealth from the future AI trillionaires.
Same. GPT-6 has been a huge breath of fresh air for me. Fixed 80% of the issues I was having with Sol. I just gave it the same task I gave to Sol a few months ago, and it knocked it out of the park comparatively.
Apple did that… eventually. Early iPhones weren’t very secure. Apple only got serious after they dominated the market and reducing the theft value of iPhones became a priority.
Even the cheapest commodity encryption chips have had this capability for well over a decade now, Unlike Apple, Flock are not a hardware pioneer, not by a long shot.
The Android documentation has an example of how to use hardware keys, and a Chinese OEM (IIRC) was found using the example key provided by Android sample code - and yet that was more effort than Flock applied, since their ARM SoC support it.
reply