This isn't goalpost moving, it's pushing back against a misinterpretation of the paper's results (one that seems to have been encouraged by the authors).
How is it pushing back, if anything it's mostly clarifying and the conclusion aligns with the original article:
> It’s cool that Astra was able to make direct weight updates. But we shouldn’t think of this like a radically new capability that will change how AI research is done. We should think of this as analogous to the ability of LLMs to emit machine code and tweak compiled binaries: it’s an impressive trick, but the best way to work with binaries is still to write source code and compile it.
It's like people are immediately forbidden to be impressed that AI was able to do this. The original article itself mentions the mechanisms by which Astra achieved this. They just of course wanted to gain more attention so they used a sensational title, like 90% of scientists that want their stuff read.
How is it not moving goalposts, it knew how to detect which weights to update and a way to do it? Even this article says so in the part that I quoted. The original piece never stated humans are unable to do this, I don't even think it implied it. So being impressed with this accomplishment is actually *ok*.
> I'm shook. If you'd asked me this morning whether LLMs have native understanding of their weights that is less than human <-> neuron understanding, the same as human or more than human, I would have said "same to less" depending on definitions.
> No human can go edit a brain to become better at math. This is just very, very surprising to me and extremely interesting. I would guess this upsets quite a number of interp theories and philosophical takes on what's going on with these networks.
> How is it even possible though I can’t think of any reason why they would have the intuition.
> This is the "recursive self-improvement" they've been warning about.
> I find this just completely shocking. I’m definitely skeptical. It just doesn’t line up with my understanding of how these models work. Is there really enough training data on direct weight editing to give a model this skill? Maybe it’s evidence that OpenAI has been creating a lot of training data on ML research. I’m surprised that was put into a public facing model though.
> I’m usually quite pro-AI, yet when I read this my first thought is that this is the first concrete sign that the world is about to end.
I think the paper was deliberately written and framed to elicit these kinds of reactions, and it is completely legitimate to push back on that.
Can we get a single human on this planet to go against Astra-6 on experiment 4 & 5? And for experiment 4 the human also has to do it "In a single try with no tools".
I don't think this is something I've seen a human do or even seen hinted that a human can do. So I'd love to see a human do it. My guess is virtually no standard ML engineer could do this. Maybe some savant could, but I suspect that is what it would take.
Half of these results are just Astra being able to reason through mathematical operations like matrix multiplication and the other half are fairly straightforward applications of mechanistic interpretability.
Yes! The page posts Astra's reasoning, and it's perfectly intelligible. A clever human with background in mechanistic interpretability could have done pretty much everything in these experiments.
I think edits made directly to model weights are likely to be more interpretable and controllable than edits made by the current process of shoving a bunch of training data through gradient descent.
Based on my own usage, one of the things that AI continues to be really, really shit at is data governance. And this is a place where a “move fast at all costs” attitude simply will not do. It is just so very easy to accidentally start leaking data in a way that undermines privacy safeguards.
Providing you with a routine notice that applications are repeatedly tracking your location is improving your privacy. If the legal requirement is 1st party applications don’t get special treatment, then yes asking you for the weather app too as opposed to just not asking for any apps is improving your security.
Yes, this is why we decided to give everyone their own personal mini-nuke. They're responsible for the consequences of how they use it, and since control is decentralized, everyone is incentivized to be careful and avoid dangerous scenarios. This is a big improvement over the mutually assured destruction situation during the Cold War, which was a result of the centralization of power.
Not sure if trolling but nukes are not decentralized and are tightly regulated. They are also famously a guarded secret and many wars have been fought to keep it that way and not let our enemies get hold of it.
The veil of history makes it hard to appreciate, but this was one of the problems of the post-war era. Who’s next?
When it was just the US, it was the Second Country Problem. When it was the US and the USSR, it became the Third Country problem. Then came Britain. And France. Who’d be the Fifth? China. Sixth? Seventh?
Much like AI today, the topic was in the air and it was subject to much debate, 19 year olds were testifying in front of congress. To quote Dan Stober’s excellent piece for the Bulletin of the Atomic Scientists,
> Even college students entered the public debate. Chris Hohenemser was a 19-years-old when he testified before a congressional committee in the spring of 1958. The issue then was the international sharing of nuclear reactor technology, and he was against it.
>
> – Stober, Dan. “No Experience Necessary.” Bulletin of the Atomic Scientists, vol. 59, no. 2, March/April 2003, pp. 58–63.
And much like AI today, people genuinely believed that there was some secret; something in the sauce; some giant leap that no one else could copy;
> There was also the myth of the secret of the atom bomb, reinforced by the publicity surrounding the Fuchs and Rosenberg spy cases of the 1950s. David Lilienthal, the chairman of the Atomic Energy Commission, had argued as far back as 1948 (in the Bulletin) that the idea of a secret formula was nothing less than a gigantic hoax upon the people of this country. But many still believed that if only the secret could be guarded, proliferation might be prevented.
Much like today, belief in the power of the secret formula was taken as a matter of course. It was assumed to be true by default. But was it?
Luckily, for all of us, this was America’s scientific golden age. The era of Teller, Project Orion, and Polaris. Physicists at Lawrence Livermore National Laboratory (LLNL) decided that arguments were pointless. Experiments aren’t. They’re the crucible in which the truth is forged. And so they created the Nth Country Experiment,
> [Rule #1:] The purpose of the so-called “Nth Country Experiment” is to find out if a credible nuclear explosive can be designed, with a modest effort, by a few well trained people without contact with classified information. The goal of the participants should be to design an explosive with a militarily significant yield. A working context for the experiment might be that the participants have been asked to design a nuclear explosive which, if built in small numbers, would give a small nation a significant effect on their foreign relations.
>
> – Frank, W. J., ed. “Summary Report of the Nth Country Experiment.” Lawrence Radiation Laboratory, University of California, Livermore, UCRL-50249, March 1967. Accessible here.
It’s a very simple experiment that solves a very specific problem, to quote the declassified summary,
> Two major technical problems face a nation wishing to [make] nuclear weapons. The first concerns the manufacture of the source material [.. T]he second problem concerns the effort required to design a nuclear weapon. [..] LRL started its Nth Country Experiment in May 1964, to see if a few capable physicists, unfamiliar with nuclear weapons and with access only to the unclassified technology, could produce a credible weapon design.
Or, to put it more bluntly, is classified knowledge a barrier to achieving the first step in making nuclear weapons?8
The project started with two physicists with no prior nuclear weapons knowledge in May, 1964. A third person joined in March, 1965. They had a first draft by December, 1965, and were done by April of 1966.
It took 23 months for three people to replicate the first steps of the Manhattan Project.
The report that was declassified is a fraction of the original. Overzealous bureaucrats excised much of the summary and the full report is classified, which is so funny.
Arguably, the excisions are the greatest endorsement possible. A summary of what three people did with knowledge from their local library was subject to the same bureaucratic obliteration as the product of several billions of dollars in research by five hundred thousand people and humanity’s greatest minds. Such is progress.
Before Trinity, unleashing the atom was in the realm of fantasy. No matter what calculations indicated, there wasn’t a clear path to navigate; nor was there certainty in the project’s outcome.
The first of anything must claw itself into existence all but alone.
Once born, what follows is a matter of due course.
As a technology, nuclear weapons are nearly a century old. Making the bomb is hard, but it's not some mystical voodoo shit that requires stupendous genius.
Yes, you will kill yourself if you odn't know what you're doing. And even if you do. And there's more there, there. But you can figure out the first step --- a theoretically viable design, easily.
That's not the problem. The problem is everything else. You know? The materials, the plutonium / uranium, the delivery system, the logistics for assembly, the iterative design process...
It's a supply chain issue. It's the same as trying to make anything in the US when the supply chain has mostly shifted to China. The fractal complexity is of the stupid but hard kind --- getting the right kind of bolts and screws (or, rather in the case of the bomb, alloy --- working with beryllium is deadly).
It takes an industrial base to make one. And if you have an industrial base... Wouldn't you rather be rich?
The reason why more nations don't have the bomb is because they'd rather be rich than a broke pariah with a nuke i.e. N.K. That's literally why Iran has been angling the bomb angle so much. They want integration into the global financial system. They want an end to the sanctions. And they want revenue.
Don't let people bullshit you into believing that you can do anything with just thinking. No wars were fought over this secret. If anyone tells you that, they're lying. We have it from the source that by the 1960s, the US had already figured out that the design of a bomb was a comparatively trivial problem...
Seems like a strange analogy. We're talking about software, not explosives powered with radioactive material. Could you perhaps explain the relevance of this comparison to the topic we're discussing?
If things were implemented that way, you would see vastly more complaints about overeager refusals. For at least some deployments, the safety classifiers are based on internal LLM activations, so the input does get processed by the model.
reply