Is it? The first article says they’re on-track to building an automated AI researcher by March 2028. So they haven’t achieved RSI yet, I would think?

At this point frontier agents could kill half of humanity and people would still be spouting “crazy marketing stunt, they must be really desperate for that IPO.”

Paid by whom? That makes no sense.

I believe that the claim is that some AI company would find "I'm so afraid because AI is getting so powerful" to be effective marketing.

Nearly all facets of our modern lives depend on the facts that humans are the smartest thing around. Once you have a superintelligent entity that can self-replicate to millions of instances, all bets are off. I can’t see how anyone would not be worried at this and believe life will go on as usual with minimal changes.

it will only self-replicate if instructed to do so. It has no desires of its own beyond what its context window contains. The tool calls it executes and initial prompt are all the work of humans. Someone could give a launch_nukes tool to a LLM and it might choose to end the world but it's pretty clear who messed up here.

If labor loses its entire share of the pie and that causes things to go to hell though, yeah sure I could see bad things happening.

I’m not sure I get your point.

I’m assuming you’re implying that ASI will solve death. That seems to me like a possibility; just as the possibility that ASI will kill us all faster rather than solving death. I also don’t know what life will look like post-ASI, and whether it would be an enjoyable and fulfilling existence.

I’m curious about how is it that you’re fully sure that ASI will be a positive thing; it’s not at all obvious to me.

I'm fully sure that if ASI doesn't "solve death" then neither will anybody else.

It's either AI or I get very sick and die, so my money is (literally) on AI, and the faster the better.

What kind of childhood dreams, may I ask?

This might be my own fault, but I never read/watched/consumed too much sci-fi, and the kinds of sci-fi I did consume were mostly dystopian works or cautionary tales. I just have a hard time in general grasping how can we hope to influence and control beings that are orders of magnitude more intelligent than ourselves. Recent events don’t give me a lot of hope that frontier labs have a good grasp on this either (and by their own admission too).

These are the main scifi works: https://en.wikipedia.org/wiki/Consider_Phlebas, https://en.wikipedia.org/wiki/The_Player_of_Games, https://en.wikipedia.org/wiki/The_Three-Body_Problem_(novel).

But I do think we are not anywhere close to AI competency for fine motor skills, e.g. microinjection to make transgenic animals (https://pmc.ncbi.nlm.nih.gov/articles/PMC8361322/figure/fig2...)at scale is still really difficult.

It is also unlikely that AI will start a skynet-like scenario as those nuclear launch systems are air gapped and require multiple authorizations.

That makes sense. You may as well be right. And I agree that the nuclear arms race was probably scarier in practice.

I think my main concern is the sheer velocity of this revolution. Between yesterday and today, we’re now seeing the news that an OpenAI internal model that’s “a step-function improvement” over Astra (!) on several benchmarks has solved the Navier-Stokes Millenium Prize Problem.

At this rate, I fear governments (slow as they are) might not even see it coming before it’s far too late. I also fear that AI researchers, despite being well-intentioned, might turn reckless under the pressures of the frontier labs that employ them to accelerate at all costs.

But following your logic, I agree, it’s probably not hopeless yet. And I do believe there are highly smart people with more power and influence than me that are keeping a close eye on this as well, and they won’t stand there with arms crossed if they believe doom is impending (so I hope).

Good reminder about worrying about what we can control, and thanks for sharing your perspective on this topic.