Roughly speaking, O(n log n) means "grows no faster than k(n log n) for some k", whereas o(n log n) means grows strictly more slowly* than k(n log n) for all* choices of k.
"The psych had told me nothing of the nightmare that dependence on those meds can create"
I hear this all the time and never get it. Didn't these people go to school? People talk about this every day on the internet. It's right there on Wikipedia, even.
They're dealing with neurotic (not being judgment, these are a powerful class of drugs) people. Like lines of code, (saying) less is more. "oh by the way, some people get how's
hooked on them and it really sucks to get off them". then the patient doesn't take them and get the relief from the problem they saw the psychiatrist for. Any responsible psychiatrist is well aware of the problem and going to get their patient past their current crisis first, and then not give them a script for 18 months for the patient to build an addiction. Unfortunately, not all psychiatrists will do that.
Yeah, that’s a classic dilemma doctors have to face. Even my gastroenterologist stopped giving me details as soon as she convinced herself my issues were due to anxiety (they were not, but that’s another story).
> History judges researchers who work with the bad guys ... To them I say: learn from history and get out now. You’ll still be ahead of the curve. The current backlash against AI may seem huge, but it’s nothing compared to what I think the backlash will be in coming years.
> I chose 7 years because I see certain parallels between the situation with researchers and Big AI today and the relationship between researchers and Epstein in 2019.
Look, I absolutely love Vi Hart's stuff. And I mean love love, and I have for almost 20 years.
But do people really believe this? That associating with AI companies, using their product, etc, is similar to taking money from a child sex trafficker?
I thought the original Mathathon sounded like a great idea. They shut that down, and I think the revised one seems like a half decent idea. But if I get involved in it, in 7 years it'll be the same as if I had been involved with Jeffrey Epstein?
I feel like the discourse on this is starting to get way out of control. I say that with no less love for Vi Hart's ideas than I have always had. But surely this technology can be used for some good?
(a) AI will replace all or nearly all human intellectual labour
(b) AI will achieve superintelligence and become a (hopefully benevolent) digital god
(c) AI will be a superweapon that defines the next era of warfare
(d) AI will render art and creativity obsolete
These are real arguments I have heard. I'm sure you have too. If any of them are true, even partly, would you really be shocked to see the people of the future looking back at us as a bunch of Chamberlains?
And would they judge us less harshly just because the people who wanted to end the economy, summon Cthulhu, build a nuke and kill art simply failed to achieve their goals? Is the goal itself not odious enough to condemn?
None of it is ridiculous in principle, but where do you draw the line? Is Google an AI company whose products I shouldn't use? What about Microsoft, where Vi worked for many years? Or is it specifically OpenAI and Anthropic? What about open source companies?
People are still arguing about capitalism vs communism, even though the laissez faire capitalism that inspired literal communism ended with the New Deal and literal communism fell some time around the Berlin Wall (give or take when China became 'state capitalism' in practice if not name).
One thing is, I am confident, going to be true regardless of AI: for as long as humans are still around, we'll be arguing about the economy and what 'good' even means in that context.
Less of a quibble:
> (b) AI will achieve superintelligence and become a (hopefully benevolent) digital god
While supremely hubristic, is that really odious given the 'hopefully benevolent'?
Like, I don't believe in any god, but is this any more odious than someone simply praying for success to a god they do believe in?
Some of the smartest people I know are also some of the people that use AI the most.
They use their time and energy on solving hard problems. Either in the abstract (research) or more practical problems (entrepreneurship).
I completely fail to see how anyone would want to equal researchers and entrepreneurs to criminals.
Plus the dozens of existing cyberattacks initiated by OpenAI and Anthropic that were simply shrugged away by their leadership.
[1]: And there's a related Tweet I can't find right now, from an AI startup founder who agrees and encourages people to start prepping supplies.
Once shit really hits the fan, I expect a good majority of people will come to view AI as a toxic force that sows discord and saps the life out of everything they once enjoyed. They'll witness endless cyber warfare, all-enveloping disinformation campaigns, devaluation of art and culture, mass unemployment and unbelievable oligarchic consolidation, spam and enshittification at unprecedented levels, incredibly complex phishing and ransom schemes, every last bit of commercial human interaction turned over to bots, complete calorie-free information overload in exchange for... what, exactly? An improved Google search that confidently tells you the wrong thing? A digital avatar that makes trip planning marginally easier?
> This is our view as well. Relatedly, average people will want to increase their level of personal resilience and preparedness as their resources allow. We will be publishing about this soon... This is why I bought a bugout property towards the end of last year.
> But do people really believe this? That associating with AI companies, using their product, etc, is similar to taking money from a child sex trafficker?
I'm not saying he did it (how would I tell?), but Altman has absolutely been accused by his own sister.
And while Musk doesn't seem to have actually gone to that island, he did seem interested.
And one of the files did show Zuckerberg and Musk dining with Epstein.
And many of the big tech leadership did openly support Trump, despite the long list of things that now includes suspicious gaps in which of the Epstein files actually got published. (Even if theres a legit reason behind what remains hidden, promising to publish then failing is at best still a huge own-goal because "they have something to hide" is necessarily true and always sounds like it means "that they themselves did").
So, yeah, I can see it being that level of bad for at least several of the big names.
(There's also a lot of distaste for Anthropic; I'm not aware of anything of this category, but even without that it is easy to tar everyone with the same brush).
This is a very loaded blog post. A unit-cost RAM is a fairly standard model of computation when working with algorithms. When we say that comparison sorting of strings takes O(n log n) time, for instance, we are also implicitly assuming a unit cost for comparing two strings. In real life, comparing two strings scales linearly with string length. Maybe that's silly, but it's par for the course. The original FFT is also assuming unit cost multiplication when we say it is O(n log n) time - we mean O(n log n) multiplies.
Similarly, yes, it's a galactic algorithm, but these things are still viewed as meaningful. The fact that it is even possible at all with a unit cost RAM asymptotically is an incredible result that nobody expected.
The string comparison model is useful in the regime where string lengths are much shorter than list lengths and can be bounded by a constant. It would be a horribly broken (i.e. unrepresentative) model if you were constructing intermediate strings whose lengths could grow arbitrarily and become much longer than list lengths.
One can of course make a puzzle out of anything, but it behooves asking what situation is being modelled by that problem.
And yes, this criticism might apply more broadly to TCS rather than just this specific artifact from OpenAI. It's reward hacking regardless of whether humans do it or LLMs.
I can't relate to this at all. AI models will surely get better at writing "enjoyable proofs," but for now the situation is what it is. You're passionate about this problem, right? But you don't want to do the work to understand the result? Fine. There's a new generation of younger, hungry mathematicians that are highly interested in figuring out why the result is true and I am sure they'd be happy to wade through it and spoon-feed you the answer instead. Maybe they should be running things.
Knowing a few PhD candidates and non-tenure track postdocs, the “younger, hungry mathematicians” are more worried about finding a damn job in the poorest job market - both academic and in industry - in a decade.
There are plenty of mathematicians who think the results are interesting. Not only do they have the "patience" you speak of, some even seem to be enjoying exploring the new results.
No idea why you're downvoted, my intial thought was that I would like to see some of this supposedly widespread sentiment as well.
I even think it's plausible a lot of mathematicians are excited by it, but the sweeping confidence of the comment you replied to without anything to back it up leaves some to be desired
Checking someone else's work carries a lot of opportunity cost, and is only fruitful if one can learn new methods which apply to the own work. This is pretty risky, especially without tenure!
The problem here is that while you can call what OpenAI does "mathematics", I would hesitate to call it science. Science as a process of acquiring and developing knowledge within a domain involves a lot more than just dumping unfinished work on the scientific community. Among other things, it involves developing frameworks and understanding of the domain, formulating questions, creating results in a fashion suitable for verification/testing/replication, and relating these results to and integrating them with that edifice.
Have you looked around recently? Because there are plenty of mathematicians that are excited to read and learn about all of the new results. They've improved on the sub-n log n result and have even made a web site to track progress on it: https://beyond-n-log-n.netlify.app/
It looks like people are enjoying themselves, having fun with the new results, and generally doing all of the things you say "science" is supposed to be about. So what's the problem?
I didn't say it is useless. Consider Ramanujan, whose work gave rise to a lot of interesting math, even (and sometimes, especially) the parts that lacked proofs or had other gaps (probably because it was obvious to him unlike us mere mortals). But a singular genius, whether a person or a machine, does not science make.
Kinda sounds like computer science vs developing - in the sense that people with a master's in CS and are dedicated to the craft will write wonderfully artistic software, while it doesn't actually take a love for the process to write code and get hired at some tech company (even less so now with agentic development).
Not what I am getting at. In fact, the problem I am getting at is the abolition of existing scientific and engineering principles and processes without a replacement and applies to programming with agents as well. This is not about artistry, but about building durable things.
The question is why the situation is what it is. Did OpenAI publish a large volume of unreadable proofs because that was their best attempt to contribute to the field of mathematics? Or does OpenAI feel that it’s more profitable for them if people come to see mathematics as something that’s less focused on understanding and more focused on using AI to generate proofs?
Even with humans, the first publication is rarely the best expression of the lesson. Getting published does more than establish priority; it also frees up the community to build on the result. Nobody expects a human to wait until the proof is comprehensible to anyone except themselves and the referees.
I don’t think that’s true? I never did research math myself, but the people I’ve known who do would definitely invest time in making a proof clearer and better even if they had the idea basically correct. There’ve been multiple recent stories of researchers saying “we’re going to publish this lame proof that isn’t up to our standards because the AI companies let us know they’re going to scoop us if we don’t”.
Mathematicians care a lot about the exposition of their ideas, invest a lot of time in giving talks, writing, and don't publish too frequently, compared to other sciences.
They normally don't feel like they are in some kind of race to publish the results ASAP and claim priority. Cases like that are very rare (but they get media coverage because they are so unusual).
OpenAI did a publicity stunt, their motivation is not to make a good contribution to the field, which has very different standards and culture, compared to the AI labs.
Yes, I guess it matters a lot whether this was quite close to the best they could do or the best they could do given specific resource constraints or whether they just didn’t bother to try doing better (eg to invest more tokens into readable papers)?
> I guess it matters a lot whether this was quite close to the best they could do
As the author of the post points out, there is no way this is “the best they could do”. It’s a write up that didn’t involve someone with the math + communication skills required to clearly explain the result.
"AI models will surely get better at writing "enjoyable proofs,"" why? Why is that surely true? they've increased in all other capacities at shocking rates while still writing awful, slippery, turgid prose. Very silly to assume that this will just go away.
I think it’s silly to ignore the sheer rate of progress that’s occurred in the last 5 years. Compare gpt 2 output to latest foundation models and then re-read your comment. You’ll realise just how out of touch you sound.
people replying to this seem to be unable to abstain from strawmanning that I am denying the rate of progress. I am not. 6 months ago they were good for an occasional technical lemma and now they can solve major open problems. The writing has only gotten worse (the papers in the big drop are completely unreadable, even to experts in the appropriate fields). If you're going to respond to a comment try to respond to what it's actually saying.
Every single day now, for close to 4 years, ever since ChatGPT 3.5 was released - there's been people dismissing AI progress. Every step of the way.
It is entirely possible that one day progress just stops or slows down, but with current evidence, I don't find that too likely - at least not in the near future. The sheer amount of resources being put into this (AI) race is mind-boggling.
So while past performance does not guarantee future results, I'm just going to kick back, and assume that many of the current issues will be fixed with future models.
The progress is clearly not uniform though, it’s shaped by how amenable things are to collecting data and verifying computation. I think writing clear and cogent prose is harder to quantify than formally verifiable logic, and this is why progress in coherent communication has been slower than raw problem solving ability
Maybe this is just a matter of what model developers choose to invest training resources in, but I don’t think it’s inevitable unless clarity is made a higher priority
only addresses it in a magical thinking "lines on a graph mean that tomorrow Claude will be so entertaining it'll make everyone on earth die in a stupor Infinite Jest style" sense. capabilities have increased while writing has either stagnated or gotten worse and more impenetrable. if you think quality of writing is going to fall to the bitter lesson that's fine you just actually have to make that argument, and how it squares with the fact that AI seem to prefer their own dross to human writing, and that RLHF is increasingly being replaced by RLAIF.
So OpenAI should be able to flood the world with AI pollution and ask scientists and mathematicians to wade through it all and tell us if there is any sense in it, then sit back and wait for them to report in?
OpenAI in fact didn't know what to do with results and didn't want to flood the world, so they asked mathematicians. Mathematicians recommended OpenAI to release them. My guess is it would have been better for OpenAI if they didn't release them. OpenAI is basically doing this as a goodwill.
People seem to have very misguided ideas about why OpenAI is doing this at all. It is not to brag or to torture mathematicians. It is an eval. OpenAI is known to be willing to pay large amount of money to get a good eval, think FrontierMath. FrontierMath is now saturated, so they need a replacement eval for math. Open math problems are actually a fairly good eval, although a proper eval is better (eg FrontierMath has known difficulty and have tiers from 1 to 4).
Mathematicians would prefer if OpenAI didn't use open math problems as an eval, but OpenAI is not obliged. I actually think OpenAI wouldn't point AI to open math problems if unsaturated FrontierMath Super Duper is available, as it just angers mathematicians, but such eval is not in fact available. Given OpenAI used open math problems as an eval, they could just throw out the result (this is in fact better as an eval since it will keep problems useful longer), but mathematicians preferred to see the result. So OpenAI released them.
As I said, a proper eval is better, but it measures something real that gives a good training signal, and there is lack of good alternatives for math eval. Since the result is 372/4000, it is also unsaturated.
I think this is right - it can be seen as trying to get free feedback from the community. In that way it’s reasonably described as exploitative, since it’s not a good faith effort. The problem of ai slop being submitted to conferences to get publication counts is similar.
"Mathematicians recommended OpenAI to release them."
This is a misleading characterisation of the mathematicians' position.
The very first paragraph of the AGMAI recommendations explicitly states:
"we do not endorse this practice, and we ask them to stop testing advanced mathematical problems on proprietary models." You appear to have acknowledged this by saying “Mathematicians would prefer if OpenAI didn't use open math problems as an eval…”.
The mathematicians did not ask OpenAI to produce these results. They explicitly asked AI labs to stop producing them in this manner. Their subsequent recommendations concern what labs should do if they have already produced significant results, not an endorsement of the practice.
Furthermore, the recommendation was not simply to release the results, but to responsibly release already existing results. Section 2.B, Step I, explicitly recommends "...labs that have AI mathematical output that is not understood by the people who prompted the AI systems", to search the literature for relevant prior work, provide appropriate attribution, and improve the exposition of AI-generated proofs before releasing them, rather than leaving this work to mathematicians afterwards.
OpenAI published the results on GitHub while still exploring repositories that meet the committee's guidelines. So they followed some of the recommendations, but not all of them and hence, did not release the results as requested by the mathematicians.
I do not think it is a settled matter whether this was done out of goodwill. This is because releasing these results as they were can benefit OpenAI more than releasing them according to the AGMAI recommendations. AGMAI recommended in section 2.B, Step 1.5 that "Each time a solution to a problem is released, it should be clearly documented how exactly AI came to be used on that particular problem. If many results are released at once, then in addition to the results themselves a further document should be written and made public that references all of the released results and explains how many other problems of comparable difficulty the models tried and failed to solve, as well as how the problems were chosen." If the results are released, it is easy to expect that the media will discuss the capabilities of the AI used in the work, as indeed happened. If this AGMAI recommendation was followed, the media would plausibly have also discussed the number of failed attempts and then the overall attitude would not be as favourable to OpenAI as it is now when it comes to the capabilities of the AI that was used. OpenAI did release on GitHub that approximately 4,000 problems were attempted and resulted in 719 manuscripts (after 3 containing suspected errors were removed by OpenAI) across 372 families of problems, but this does not give a calculable number of problems it failed to solve. I do not claim to know OpenAI's intentions or reasoning when these results were released and am not arguing that it was done with improper intentions, only that whether it was done out of goodwill is not a settled matter.
AGMAI's October 6 statement explicitly clarified that its advisory role should not be interpreted as an endorsement of OpenAI's process, and that it was up to the mathematical community to assess how successfully its recommendations had been followed.
Recommending how to responsibly handle the outcomes of something you oppose is not the same as asking for it to happen.
It is also worth adding that the Association for Human Mathematics published a statement (which Tao reposted on his blog) in which they explicitly say the following:
> Mathematicians did not ask for this work to be done. The Advisory Group on Mathematics and Artificial Intelligence, from whom OpenAI has claimed to derive its legitimacy, opened their initial advisory statement by saying that frontier AI corporations should not test advanced mathematical problems on internal models. In ignoring the central premise of the Advisory Group’s position, OpenAI has indicated total disregard for the norms of scientific research — norms that guarantee that mathematics remains trustworthy, ethically researched, and in the public interest.
>> I do not claim to know OpenAI's intentions or reasoning when these results were released and am not arguing that it was done with improper intentions, only that whether it was done out of goodwill is not a settled matter.
That's right honourable of you but for me it is very clear that the only incentive in AI companies' effort to produce mathematical results is to advertise their technology. There is no reason at all to assume they have any other motive; certainly not any kind of interest in mathematics as such.
Yes, they should. They have invented a magic button that can tell you the long-awaited answers to the burning mathematical questions that you've spent your life researching. The caveat is that the technology is still new, so the explanations "are not fun to read" like set theory papers usually are (lol). If you don't think that's a worthwhile tradeoff, that's your call, but it sure as hell isn't everyone's.
I don't think the claim is "they definitely have an oracle that solves the problem, and I reject it because it's hard to read". The claim is "OpenAI claims to have used an oracle to solve the problem. The proof is very difficult to read, and to even know if it does or not, we have to go through it with a fine-toothed comb, but they're going around claiming they definitely solved the problem (or at least getting press that claims that which they aren't pushing back against) and this might convince the people who sign grants even if it isn't true"
People who are invested in the idea that we've invented a general intelligence, now, which includes all these companies that are literally financially invested in this claim they are making, will tend to believe that its results can already be trusted in domains like this. Some mathematicians seem to believe some of the proofs written by their models, and some, like this one, don't. I do think it's valid for an expert to push back against the claim that the best use of their time right now is to verify the poorly written work of everyone who's claimed to solve the problem
Over the year, nothing they've released with a lean proof attached has turned out false (That's sort of the entire point. It's not impossible but it's really difficult). There's a reason most mathematicians, including the ones vehemently against OpenAI's dumping are not arguing the results are secretly false or have a high potential to be. And indeed, if that were the case, it would quickly become apparent and all this worry about grant signers would vanish into the wind. It's very easy to ignore nonsense. The problem is that it isn't nonsense.
> Over the year, nothing they've released with a lean proof attached has turned out false
That's not true. [0]
> On July 25, Ramana Kumar published a repository containing a sorry-free "disproof" of the Collatz conjecture, produced with AI assistance. It is not a valid proof because it exploits a bug in the kernel's handling of nested inductive types.
Even in this dump we're talking about, it hasn't been true. [1]
> In “Algebraicity of Weil classes on split abelian eightfolds” a sign error invalidates a stabilization-trace cancellation argument and the construction used by two dependent papers.
1. That wasn't from Open AI or any major lab. I don't know what random people are getting to. It's curious also that it had no natural language proof attached. Usually these labs have a NL proof then translate to lean. Lot less possibility of lean maxxing.
2. None of the results Open AI retracted had an attached lean proof
I really don't know enough about it to know whether you're right or not, nor do I know whether or not you know enough to make the claim you're making, so I won't make an argument one way or another because it's non-sequitur to what I said anyway. The fact that you or I or Sam Altman or Terrence Tao believe the claim is irrelevant to whether this obligates the person who wrote the blog post to believe the claim, and it sounds like he's willing to consider the possibility that it is right, and would read the paper if it reached a threshold of comprehensibility expected of people making that kind of claim.
I's not a non sequitor because it cuts right to the point. He's under no obligation to read it sure, but that doesn't mean Open AI isn't justified in claiming to have proved it. The justification isn't Sam Altman's belief or Tao's or anyone else's authority. Results accompanied with lean-verified proofs whose formal statements match the problem at hand have arguably stronger justification than the vast majority of human math publications.
The problem isn't just that the papers aren't fun to read. The problem is that a lot of the research that goes into solving these issues leads to other discovers, new fields to explore and people have to develop new approaches to solve them. The other part of it is, the quality and the enjoyment of working on these problems leads people to find new and other interest problems to work on.
If you just strip mine the answers and Sam Altmans magic button solves 100/100 problems, what's next? Who is left to come up with a new interesting question for the magic button to solve?
Lastly, life and the present moment is all there is, if there is no enjoyment in anything we do, then what's the point of all the "living for ever" Altman et al want to achieve.
We will live forever to read boring papers generated by LLMs? Literally sounds like an eternal hell.
People are already finding stuff in the release to get excited about, and as the models get better at distilling proofs to make them more coherent, this will only amplify. Of all the things to worry about, human curiosity and the ability to run with new ideas probably aren't at stake.
In fact, I can't remember a time when I was more excited about the future of science. This could herald an end to the replication crisis, and kill off bullshit science completely. The danger of course is that we end up with two companies effectively dominating cutting edge research in every field, but it remains to be seen if that's even possible given the pace of improvement in open weight models.
Nuclear fusion is already proven by the universe to be a viable energy source by the fact that the sun exists but people still work on understanding and taking it and developing new approaches to accomplish it. People didn't stop experimenting with and developing programming languages because technically they're all turning complete and the first one was "enough." Y'all will be fine - every JavaScript framework that exists is someone looking at a theoretically correct and complete solution and deciding actually it sucks and they could do better. "I want to understand xyz but the proof is trash and I think it's ugly" will be plenty motivation for a lot of people to work on it.
You don't have to wade through the slop. The point is a giant star in the sky figured it out and that doesn't demotivate you from figuring it out yourself
He's tracking the community progress on sub-n log n multiplication. OpenAI started with 1 - 1.63e-55. The result has been now improved on 115 times, and the current record is "rohanarun"'s 1 - 9.87e-5. I'm sure by tomorrow it'll have improved again.
Does this look like people aren't having fun? Does it look like they aren't discovering stuff? It looks like it's spurred a cascade of interesting community activity. It doesn't really seem much different from what happened with the twin primes conjecture. Isn't that supposed to be the point of all this?
It won't be like that, it will just divide the scientific community. The vibers and the people who do the work.
I see it in my job now, there are people who do the work and know what's going on, the people who actual get shit done. Then there are the people who just leach of everyone else, vibe and have a bunch of prototypes adding no real value.
I don't understand. Why is this the onus of scientists and PhDs to review whatever results OpenAI had dumped out? If OpenAI had produced incomprehensible papers, surely any journals would just reject it, or demand the author to do a complete rewrite? Unless we are talking about a race to solve problems, which PhDs are afraid that they had been scooped up on?
They should certainly be allowed to share their findings. No one is forcing scientists and mathematicians to review the findings in general. It’s just the case that the findings are of such such a quality that it would not make sense to ignore them wholesale
> No one is forcing scientists and mathematicians to review the findings in general.
This is like "no one is forcing software engineers to use AI tooling" or "no one is forcing you to show your ID in the airport" or "no one is forcing you to own a car in your small midwestern city" - there can be no law requiring something and the practical consequences of not doing so can be so painful that you're effectively forced anyway.
That is exactly what I’m trying to say. The findings are of such such a quality that it would not make sense to ignore them wholesale.
That’s why it doesn’t make sense to present AI companies as dumping or burdening the scientific community into doing labor for them; the scientific community is self motivated to do so.
It's not self motivated. The motivation is not "this is doing amazing things for us", it's "if we don't review this, the bullshit headline complex and the bullshit-spewing (sorry, marketing) departments of tech giants are going to misinterpret/misrepresent everything and our grant money will be taken away".
The best part is that when the academics fix OAI’s issues, the model gets better and OAI shareholders get richer and more powerful!
As someone who uses LLM tech occasionally, this is why I prefer using open local models. If I’m making myself obsolete, at least I’m not making some asshole richer and their closed model better.
You know, I kinda relate to the feeling of not wanting look at those outputs if I think of it from a layman's perspective.
I just imagined that instead of math papers, they released 700+ feature length films, and the only way to tell if one of them is any good is to watch it in its entirety.
That feels pretty unappealing to me.
I know it's the same for human made films, so what's the difference right? But those are good enough most of the time that it's a decent bet, and the people that made them had real skin in the game.
Contrast that with something made by a nondeterministic slop machine with no skin in the game where small details can be off in a way that's jarring. Right out the gate I have an aversion to committing that much time to something that very well may waste it.
That's actually a really interesting thought. Given Sora, and the amount of funding they have, they could have created started their own film festival and dropped 700+ feature length films, had they wanted to go in that direction. But they didn't. Hmm.
What else should they do? See these models get smarter and smarter, somewhat-solve things but only to the tune of 90% what mathematicians (or experts in any other field) would deem acceptable, and then gate keep the findings for the next few years going through peer review and paywalled journals? I for one welcome the flood, bring on more in every possible industry and see where all that progress lands up. Sure it will upset a lot. A lot of things also upset the luddites.
Are you going to be doing the work to verify the results ? Will you just expecting other people to wade through the flood and reap the benefits later on?
Nobody is forced to verify the results. I am honestly not expecting anything other than AI to get smarter and smarter and people who are motivated and interested enough to pick up after it; and potentially reap all the long term benefits ahead of those who aren’t (seeing this happening with software development in my own field). But in the end; if you don’t like it, you’re not forced to do anything.
> this is an opportunity for people to pick up where the model left off and run with it
Like people enjoy racing in front of a stopped train? As soon as they turn on the engine again, they will run you over. The questions that remain will be only the low value ones, not worth the effort to vacuum up.
So no, the smarter, hungrier people are not the ones that are going to swoop in. It will be the most desperate.
> Some output is going to be wrong or incomplete
This is a very human take on the situation. No, the Lean proof is not going to be wrong, and it will be incomplete only in the sense that OpenAI didn’t try to push the results further.
> There's a new generation of younger, hungry mathematicians that are highly interested in figuring out why the result is true
The problem is that right now mathematicians don't have the economical incentive to read these AI generated results. Even if you love mathematics and all that, it's always more important to get a job, and for that it doesn't seem like a good idea to invest time around problems that AI touches because you can't compete with it and you don't know if tomorrow they'll improve by x10 the sota.
I am a mathematician, and I do have the incentive to read the results. Two results in the drop were two major life goals of mine, and all I got was three lousy citations. :) But a third question I've spent a lot of time on is a not-so-hard consequence of one of the lemmas in there. So, yeah, I do have the incentive.
Of course, it's not clear at this point whether reporting such a result even matters, but still. In its own right, it's a very cool result.
Of course, everyone is curious about these results, we're mathematicians after all. But by incentives I refer to those things that really sustain the system: obtaining permanent positions, grants for future projects, ideas for PhD theses, etc.
The future is uncertain, so the safe thing to do is write it up if we discover some useful connection between our research and the openai results. Maybe it matters for getting positions grants etc, maybe not. Ideas for new results: the openai drop is full of those.
Maybe I misunderstood the question. I'll give it a second go, maybe I'll fail again. Feel free to reformulate.
The term SOTA suggest that you approach this from a perspective other than a pure mathematician's, maybe ML or computational complexity? The openai results help me answer other open questions. It's a yes/no thing: is question answered or not? It's not a question of "okay, but will AI win back the SOTA from me again?"
The statements of the lemmas are quite clear. We can write it up if we spot that they have important consequences. The proofs are impenetrable to me, for three reasons: they are very complex, they are badly structured, and they use math unfamiliar to me. So I haven't looked too deeply yet. Maybe when they get their autoformalization stamps of approval.
I think quite a few mathematicians would be interested in using AI to figure out the answer.
But providing the answer in gibberish along with a certificate is not that, it's at best a cruel way to do it, but I'm leaning towards the idea that it's a fundamental misunderstanding of what it means to do math and what it means to communicate a result.
If you think sending an answer in gibberish is acceptable just because it's true then SSdtIG5vdCBzdXJlIHdoYXQgdG8gdGVsbCB5b3UsIGJ1dCB3ZSBkaXNhZ3JlZSBvbiB0aGF0.
I think this might feel about the same as getting a PR from Claude that purports to solve some issue that it deems exists, but it doesn’t conform to the contribution guide, isn’t clear in its objectives and looks quite likely to be utter bullshit. I close them without comment and lock the issue.
There's a "Silicon Valley-ism" for you. We offer a thing in whatever form we want and people "who are passionate" will gobble it up, should gobble it up, 'cause they're "passionate".
You don't know what you're talking about. Just because it was created by an LLM, and verified in Lean, does not make it true. The whole point of writing a proof is for it to be understandable.
Suppose I directed some llm agents to factor the primes from network logs of your machine. I then publish the private and public key in full. You would not change your keys of course because its not true right?
> I think the next step is to demand that proofs either be human-scale or they prove that a human-scale proof is impossible and the machine proof is as good as it gets.
Who do we demand this from? The AI companies? Or the mathematicians who are worried they will have nothing left to do?
Aside from the usual squabbling about AI, it seems the bombshell claim is this:
"In particular, we show that the formalised Lean proof does not correspond to the NL proof of blow-up of solutions to the Navier-Stokes equations."
So these authors seem to be claiming that OpenAI has not really proven Navier-Stokes at all. If I get their idea correctly, they are claiming that the LLM has not formalized the original "natural language" idea of Navier-Stokes correctly. If true, it would mean that their purported Lean proof is not actually a proof of Navier-Stokes at all, but something that is an incorrect translation of the original natural language idea. If correct, this is a really bold claim and I would like to see if other researchers agree.
No one is disputing that the Lean formaization of Navier-Stokes is correct, so we should have high confidence that the generated Lean proof is valid.
The authors are claiming that the Lean proof is not the same proof as the NL one. Therefore, we shouldn't yet have confidence that the NL proof is valid.
This is an important claim which the math community will need to work through. However, the Lean proof alone is sufficient for OpenAI to (reasonably confidently, leaving aside questions of academic manners) claim to have proven NS.
So, the AI wrote a NL proof of Navier-Stokes, then incorrectly auto-formalised it to Lean, but still ended up with a verifiable proof of Navier-Stokes? That seems... strange?
This same thing happened back in the 10 advances in math and CS release a month or two ago. The non sofic group construction relied on false prior literature. They realized this and fixed it in the lean program but didn't modify it in the writeup, so the written proof was both incorrect as stated and did not correspond to the lean proof. I was surprised how little press it got at the time, it seems like a huge risk factor.
Or the AI wrote a NL proof of Navier-Stokes, began rewrite in Lean, then discovered a false / handwavy / easier to write in Lean / etc approach of some parts of the proof, and modified it accordingly. Since there wasn't any backpass from Lean to NL to include any changes it did due to any of the above reasons, the proofs aren't identical. That's what I think is most likely.
If the reason for the differences was done intentionally in Lean (as opposed to hallucinate e.g. m+4 vs m+5 as mentioned in remark 3.2), then a simple recording of differences, and then afterwards pass back any changes to the original NL would fix the issue. If it was hallucinated, then there is no guarantee it wouldn't keep hallucinating, and thus you might never end up with the same proof no matter how many passes you do back and forth (see remark 3.4).
I don't know if we can really be clear about the order of things, but I think even without AI maths is filled with "someone provides a proof of X, and the proof itself is wrong/incomplete but X itself is true".
"Incomplete" proofs might be a way of viewing this. You have a NL argument to prove X. It turns out the NL proof has holes you can drive a truck through. So... you go around and patch the holes.
The resulting proof is different! You can start off with a bad proof and find a correct proof. Sometimes.
EDIT: for French speakers (maybe autodub gets you there) I saw a very nice simple case of this recently. A commonly stated proof for a relatively simple theory. The proof has a giant hole in it, and completing it requires some work [0])
e.g. one of the lemmas in the NL description is false, but a weaker version of the lemma (that does hold) is sufficient for the proof, so the (incorrect) lemma is never formalized.
It's not actually the same model that solved the problem that did the translation. Astra did the translation after the intenral model produced the NL Proof. As for the discrepancies, It's not necessarily right to think of this as 'incorrect formalisation'. Maybe it was essentially a 'proof refactoring'. Maybe Astra thought some parts could be easier expressed in a certain way, or maybe aspects of the NL proof were kind of handwavey etc.
Why do you assume this order? I would assume the AI starts with lean (that's what it was RL'd on at least) then tried to translate the proof for us humans, which is quite hard.
I'm not sure about it but it seems plausible given the potential error.
Edit: they explicite say that your order is correct in the post
I would expect it's the other way around. The AI wrote a Proof of NS in lean and write up a NL proof based on the lean one. The authors claim that openAI did NL -> Lean, but that is unsubstantiated.
> The agents arrived at their resolution on Saturday, September 5, about 88 hours after the first agents were launched. Lean formalization and verification took an additional 17 hours via GPT‑6 Astra.
This seems pretty clear that they found the proof first and then mechanized it afterwards.
Trying to map this observation to code and appreciate them starting with very simple examples (that Astra screw-fixed into lean.)
But in a simplified way this is close to promting the model to create a set with the members 1,2,1,5 and it correctly creates {1,2,5} in code. So The initial "proof" / instruction was wrong and it silently fixed that. ( Which is one of the failure modes they're describing)
Proof steps can become different in Lean but still work to prove the theorem. Incorrect proof steps in the NL paper could also be converted to correct ones in Lean. The paper shows some examples of this. So the NL paper can't be trusted even if Lean verification passes, but it doesn't immediately mean that the NL proof is wrong or that Lean is checking the wrong thing.
Or maybe the AI didn’t write Lean proof at all, or rather, not the LLM at least. But instead OpenAI has an internal traditional reinforcement model that is able to stumble on the Lean proof by the share amount of compute power available to them thousand monkeys on a thousand typewriter style. And then pretend LLM did it because that is what they are selling.
I don’t. But I do know how the scientific method works, and OpenAI’s display is anything but. Until what they have demonstrated is reproduced I take their claims to be nothing but marketing. A for profit company will lie in order to maximize their profits. Above I presented an alternative hypothesis, which is probably wrong, but until OpenAI’s results are replicated I will believe my alternative hypothesis just as much as I believes the claims of the for profit company making them.
In the absence of evidence, a guess is the best you got.
And, no until OpenAI’s results have been replicated, I don‘t believe companies own words about their own products, and therefor a marketing statement is not evidence.
It's the method they used to create the proof that's in question here. They claim their amazing product did it, and therefore you should buy their product because will do amazing things for you too!
>No one is disputing that the Lean formaization of Navier-Stokes is correct, so we should have high confidence that the generated Lean proof is valid.
These authors don't seem to be disputing that this Lean formalization of Navier-Stokes is correct. I don't think that gives us any new information about whether the generated Lean proof is or isn't a valid proof of this N-S blowup thing.
> No one is disputing that the Lean formalization of Navier-Stokes is correct, so we should have high confidence that the generated Lean proof is valid.
Why? It is a correct proof. Is it a correct (dis-)proof of NS? These statements do not seem particularly related. People, including (especially?) mathematicians, find NL proofs far easier to read than Lean proofs. I would bet the ratio of people who read (part of) the NL proof vs the Lean proof is at least 100 to 1. It seems incredibly unlikely that anyone has read and reviewed the whole Lean proof (including the authors of the OP).
Is it particularly easy to see that the Lean proof does actually disprove NS, rather than proving something similar but not the same?
Well the theorum statement (what it proves) is short and easy to verify. I would not be surprised if hundreds of mathematicians have looked at it at this point.
The thing is that Navier-Stokes has a definition split off separate from the formalization, and that is what has been completed. People have looked at the definition of the final statement. This paper only mentions the proof and intermediate statement, not the final statement. The most likely case to me is that intermediate statements do not match, but the end result still holds.
Seems kinda odd then that it didn't occur to OpenAI to iterate until they reached a fixedpoint for both the informal & formal development b/c it's obvious that correspondence should have been part of their training pipeline.
Jesus Christ, so many people here who have no clue what they are talking about.
A proof of a theorem is different from the statement of the theorem. OpenAI has a Lean proof of the statement. That is all they need. There may be many different proofs of this statement, including NL proofs. It does not matter that these NL proofs may or may not be different from the Lean proof, at least for the correctness of the Lean proof. But of course the NL proof may be wrong. But who cares?
It reminds me of how provably secure software was all the rage for awhile. Until people found that the idealized system/lemmas were so far from reality that the proved security was worse than meaningless because it gave a false sense of security.
In order to prove security, you must first simulate the universe.
Why would they do that, knowing that the one known to be correct is the Lean one? Just to claim that the (correct) Lean proof did not translate well to English? That would be weak, and a colossal waste of energy and time.
Why do people program in Python instead of writing machine code?
Why are review papers published? Executive summaries? "Introduction to X" books?
People's time and computational resources are finite. Summarising information -- ideally in structured ways that preserve important properties, but even in informal, unstructured ways -- is critical for making any kind of progress in this world.
The statement written in Lean, is not actually a correct description of the Navier-Stokes problem. It's some other easier statement. That's why the Lean code may not be a proof.
Here's a quote from the paper.
"3.1. When the NL paper declares stronger statements than what Lean proves
We commence with an example that complements those in §2. In the following example the AI autoformalisation results in a different Lean proof of a weaker statement."
"Tracing the proof of (3.3) we find that the series arises from applying the Lean theorem coefficient_seminorm_bound, just as Figure 3 mentions. Consequently, the Lean results discussed in this section are weaker than (3.1) in the NL proof."
So the question is, was the Navier-Stokes problem framed properly in the lean code, or is some easier problem represented in the Lean code?
Here is their remark.
"Remark 3.4 (Further potential mistranslations of the Navier-Stokes proof). The above examples require careful manual checking of both the NL proof as well as the Lean proof, which is delicate and highly time consuming. Moreover, the fact that we display only two examples does not mean that these are the only cases of mistranslations."
The "statements" here are statements of lemmas, not the statement of the main theorem. The main theorem statement was written by humans (prior to the OpenAI work), so there shouldn't be any concern that an AI mistranslated it.
That could be that these examples are lemmas, but in the introduction the authors conclude,
"In all cases the correctness of the formal proof does not say anything about the correctness of the NL proof."
The Clay Mathematics Institute has not yet accepted the proof as it takes a while to review.
"The Clay Mathematics Institute is dedicated to furthering the beauty, power and universality of mathematical thought. Curating the Millennium Prize Problems is one of the most important ways in which it pursues this goal. The rules governing the prizes describe the process for evaluating what has been achieved and for assigning credit. The process is deliberately unhurried, but we will provide updates."
We can see therefore that we don't know if the OpenAI program wrote the NL statement or proof correctly. Or if maybe it needs work. Or if maybe it has significant errors.
We also don't know if Astra properly wrote the initial statement of Navier-Stokes correctly in Lean.
My point is that Astra did not write the statement of Navier-Stokes in Lean, there was already a human-written statement, so there should be no concern that it has errors.
This article shows that we can't conclude from the working Lean proof that the NL proof is also correct, because they are different in some places. On the other hand I think it's unlikely that it's "very" broken. OpenAI first spent a lot of computer time to find the NL proof, and then less time to translate it into Lean. So if there were errors in the NL proof it seems they were not fatal, the agent translating it into Lean was able to patch them up as it went along.
Everyone. I don't think many people working in fluid dynamics were surprised you can find a blow-up in Navier-Stokes. What would advance human knowledge is understanding the situations in which a blow-up might occur. In that context, the lean proof is necessary, but the non-lean proof is more important.
> Jesus Christ, so many people here who have no clue what they are talking about.
Indeed. If only some of those people would see the irony.
What matters most of all, as any first year student of mathematics would know, is whether the formal problem statement corresponds to the NL statement. TFA specifically states that at least some of the allegedly proven formal statements DO NOT.
No. What the paper says is that in principle, translating NL statements to Lean statements is hard. Nobody doubts that, translating informal to formal text cannot be formally proven correct, so...
Does the paper give a single example of one of the OpenAI solved theorems with a Lean certificate where the Lean statement does not correspond to the actual statement from the mathematical literature? I don't think so, but in case I am wrong, feel free to provide that example.
> To demonstrate the effect of this result we provide
several examples of AI mistranslations of NL statements and proofs into Lean in practice, resulting in mismatches between NL proofs and their Lean ‘verifications’. These include OpenAI’s announced Navier-Stokes proof.
Could /I/ be mistranslating the paper’s formal statement to NL? I don’t think so, but in case I am wrong, feel free to cite the correct formal statement that they claim as divergent between Lean and NL formulations by OAI.
Yes, you are misunderstanding what the paper claims. Navier Stokes for example is not such an example, only the proof is different. For their other examples, none of them they claim that they concern actually the open ai solved theorems. You are welcome.
And yet I cited a specific statement made concerning N-S specifically, whereas all you’ve done is make patronizing remarks, and strawman arguments.
Have you actually read it? They give a handful of examples of mistranslations of both claims and proofs thereof, in relation to NS and Euler, though I do concede that they do not go as far as claiming outright the NS statement itself is mistranslated.
Unfortunately, lean proof alone is not enough. sidestepping the raging discussion regarding the meaning of mathematics, just because the lean compiles is not proof in itself that it is correct in the sense that mathematicians mean. Unless of course, you can prove that lean itself is correct, which you can’t.
we have seen “proofs” earlier this year that essentially abused some bugs in the kernel. it would be very convenient if every program written in rust was automatically correct if compiles - something i strongly suspect you believe - unfortunately, this is not the case for either.
In case you are not aware, the actual theorem statement of N-S was never translated by an LLM, but was written independently by formal conjectures, as they say in the README [1].
A Lean proof has a much higher probability of being correct (in my opinion) than any published (either preprint or peer-reviewed) paper, yet no one before LLMs were walking around claiming every result published can't be trusted yet (without an actual reason).
We have seen one such instance of Lean bugs, which was found adversely against Lean (as in find bug then use this bug to prove Collatz, not just found when being asked to prove it).
It's also worth to note that the way N-S (and all the other proofs by OpenAI etc) have been found is first prove it in NL then translate to Lean. I.e. it would have to first believe it found a correct proof in NL, and then afterwards either accidentally or on purpose use a Lean kernel bug.
edit:
Probably also worth to mention that the proof have been checked both by the Lean kernel and the independent nanoda kernel, so it would need to exploit bug(s) from both.
2. No exploit of a kernel soundness bug in Lean (and the independent kernel nanodo, which OpenAI also checks against).
In particular, if you trust this, then you don't need to care about anything else the Lean program does, no matter how many lines, lemmas etc. it makes along the way.
As I said above, the theorem statement have been written independently by formal conjectures, and you are free to read it yourself (or trust other people have done it).
So, assuming you don't disagree the theorem statement have been written correctly, you pretty much need to believe 2 is false, if you don't trust the proof [1]. And that's what I find has a rather low probability personally.
[1] As the book says, you also need to trust the hardware, firmware etc.
You may be overconfident here. The NL paper may be properly stating the Navier Stokes problem, and the Lean code may not be.
Here's from the paper--
3.1. When the NL paper declares stronger statements than what Lean proves
We commence with an example that complements those in §2. In the following example the AI autoformalisation results in a different Lean proof of a weaker statement.
None of the examples are of the NS lean statement. The validity of the latter is not under question ( in this paper ). Please note the distinction between “statement of NS” and “statement of lemmas used in the NL proof of NS”.
3) The comments promise functionality the code does not implement.
The first post was "It doesn't meet the business requirements the comments indicated, so the software is broken." The replying post was "Actually, the business requirements were faithfully turned into tests, so the software is fine. The comments need to be rewritten"
They published both a natural language proof and a lean proof. This paper says the NL proof does not match does not match the lean proof, using different arguments and some stronger claims. It does not claim the lean formulation of NS or the rest of the lean proof is invalid.
"The Lean code merely tells us that the theorem is correct, yet the NL proof with intermediate arguments, lemmas and results may be incorrect or have incorrect arguments."
You didn't answer the question. Quoting from upthread:
A: "In particular, we show that the formalised Lean proof does not correspond to the NL proof of blow-up of solutions to the Navier-Stokes equations."
B: "No, they're not claiming that [...] The authors are claiming that the Lean proof is not the same proof as the NL one"
The question was: what's the difference between A and B?
Your answer to that question was "This paper says the NL proof does not match does not match the lean proof".
It seems to me like A is the exact same thing as B, and you are here claiming that they're different things while merely repeating the same claim that was already claimed in both A and B.
Statement A is upthread quoting the paper, and is not the statement B is arguing against. B is arguing against the incorrect statement "So these authors seem to be claiming that OpenAI has not really proven Navier-Stokes at all."
This doesn't answer the question. The question was: what was the difference in A and B. Your answer looks like this:
A: The English language description doesn't do the same thing as the code the AI wrote.
B: The English language description doesn't do the same thing as the code the AI wrote.
A and B are the same; there is no difference. Is this what you meant? Or did you mean the opposite of this? If you meant the opposite of this, what exactly is the difference?
Which is exactly why you want a Lean proof in the first place!
NL proofs rely on not-entirely-correct prior art, and contain countless holes both big and small. When formalizing human NL proofs it is probably very normal that the Lean proofs doesn't quite follow the NL proof.
Lean is a theorem prover. If it "compiles", and the proposition as coded in Lean is equivalent to what the theorem proposes in natural language, the theorem is correct. https://en.wikipedia.org/wiki/Lean_(proof_assistant)
I don't understand then, because it doesn't sound like you understand how Lean works.
The actual proposition of the theorem is very short in Lean, and at this point has been looked at by probably hundreds of mathematicians (including the authors of the article). All that needs to be confirmed manually is that the proposition statement matches what people say the theorem actually proves. That has been done.
At that point, the rest of the actual proof steps are automatically verified by Lean - that's literally the entire point of Lean. There is no additional verification step - "What has shown the Lean proof to be correct" is the fact that the Lean program automatically validated it and generated a certificate.
But what mathematicians really care about is the actual steps. What the authors of the article have shown is just because you have a Lean program it doesn't mean the steps of the natural language theorem have been "autoformalized", because there is no guarantee the steps from the natural language description match the Lean code.
A Lean program that compiles proves what it states. The question is whether the program states what we think it does. If I recall, this program is 12 million lines long. Mathlib is not perfect, and it's not hard at all to have a Lean statement that isn't what you think it is.
So my question is whether people have actually checked that the generated program, all 12 million lines of it, has the statements that are expected.
But that's where you're wrong. All 12 million lines do not need to be checked (again, that's literally the entire purpose of Lean). All that needs to be checked is that the proposition of the theorem is what we think it is, and that has been done. What Lean guarantees is that if you start with a set of axioms -> whole bunch of steps that nobody really ever needs to read -> final theorem proposition, that those steps do successfully prove the theorem.
You say "it's not hard to have a Lean statement that isn't what you think it is", and that's true - indeed, what the entire paper is about is that the intermediate steps in Lean don't match the natural language description. But the statement of the proof in Lean (which again is quite short) is what we think it is, and hence the Lean proof is valid.
That's extremely hard to believe. First, one question arises: what are the 12 million lines doing if they're not needed? Secondly, don't those 12 million lines build up the theory that is eventually proved? Mistakes could be there that makes the outer statement seem like something it isn't, could there not be?
There's maybe an even simpler way of illustrating this.
Fermat's Last Theorem is very easy to state. There are no positive integers a, b, and c such that a^n + b^n = c^n. This likewise easy to write in Lean. The statement of the theorem is exactly one line of code.
theorem fermat_last_theorem (n : ℕ) (n_ge_3 : 3 ≤ n) (a b c : ℕ) (a_is_positive : 0 < a) (b_is_positive : 0 < b) (c_is_positive : 0 < c) : a ^ n + b ^ n ≠ c ^ n
Kevin Buzzard's team of human mathematicians attempted to prove this in Lean. They got to about 60000 lines of Lean code and were not anywhere close to finishing the job.
Anthropic came around with a Claude model that proved it in 13 million lines of code. (Buzzard's team might have ultimately used within the same order of magnitude if they finished. It's very unclear. They were at minimum years away.)
Whether 60000 or 13 million, only one line of code was necessary to manually verify whether they had successfully proved Fermat's Last Theorem, just the statement of `fermat_last_theorem`. None of the rest of the code built up the theory at all.
Indeed if those lines of code built up the theory, that would suggest that the statement of Fermat's Last Theorem is incomprehensible, if professional mathematicians cannot define Fermat's Last Theorem in even 60000 lines of code! But that's pretty absurd. You can explain the statement of Fermat's Last Theorem, completely rigorously, to a middle schooler.
> First, one question arises: what are the 12 million lines doing if they're not needed? Secondly, don't those 12 million lines build up the theory that is eventually proved? Mistakes could be there that makes the outer statement seem like something it isn't, could there not be?
This seems to reflect an incorrect understanding of formal mathematics.
Formal mathematics can be thought of as a game with rules where a statement to be proved is some game end state. If you play 12 million valid moves and arrive at the desired end state then you have proved that the statement is valid. If your logical system is sound (e.g. first order logic), then you have proved your statement is true.
The entire point of the formalization of logic and mathematics through the 19th and 20th centuries was to unlock this capability. Such that, if you only care about whether your statement is true or not and not why it's true, the intermediate steps are entirely unnecessary.
To answer each of your questions:
> First, one question arises: what are the 12 million lines doing if they're not needed?
They are the intermediate computations necessary to show that the final statement is valid. This is much like intermediate arithmetic operations in a large complicated arithmetic expression. As long as you believe that the computer can calculate each individual arithmetic expression correctly, you don't need to manually review the billions of intermediate calculations that result in the final number.
> Secondly, don't those 12 million lines build up the theory that is eventually proved?
No. Not in the way you're thinking of. It seems you are using "theory" to mean the concepts necessary to express the statement itself. This is a very different set of lines than the proof itself (and generally much much more compact). Very difficult to prove statements can be stated in very simple language. Often the theory is <0.01% of the total lines of code.
> Mistakes could be there that makes the outer statement seem like something it isn't, could there not be?
No. Only the language used in the final statement can cause this issue, i.e. the theory. The intermediate 12 million lines cannot affect this, assuming your computer can carry out arithmetic correctly.
Another way of thinking about it is that the big breakthrough of 19th and 20th century mathematical logic is to conclusively show that formal mathematics is fundamentally just a set of arithmetic operations.
Hence in a very real sense, proving a mathematical statement is exactly identical to finding x in an arithmetic expression f with some constant k such that f(x) = k, where k is determined by the statement you wish to prove (not the 12 million lines).
Showing that x causes f(x) to equal k is exactly equivalent to proving the mathematical statement associated with k.
The 12 million lines are an alternative notation, no different in spirit to decimal notation or binary notation, of writing down x. It is some gigantic natural number.
You must understand k. If you use the wrong k, then you have proved another statement than what you intended. This is usually about a dozen lines of code.
You must be confident in the correctness of f. This corresponds to the believing that your type checker (since Lean uses dependent typing as its method of proof checking) has no fundamental soundness bugs. This can be controversial and indeed Lean's type checker has had soundness bugs in the past! But crucially f does not depend on k or x. Every time you fix a bug in f you can rerun it for any k or x you care about.
But you don't have to actually read all the digits of x, in whatever notation you care about. You can copy paste it in from some gigantic file handed to you by a stranger. As long as f(x) = k, you know that you have proved the mathematical statement associated with k.
Overall, this is roughly like asking why you trust a computer to solve a long division problem that has 12 million intermediate steps if you don't actually read all 12 million intermediate steps yourself. Well I trust the computer to do one step of division correctly and to be able to repeat that faithfully 12 million times. Hence I trust the computer can solve a long division problem involving millions of digits, even if I don't actually look at all 12 million intermediate computations myself.
My understanding is that the statements in the Lean proof might not actually correspond to the Navier-Stokes problem. And that requires mathematical insight to be able to verify, which could take quite a while.
If I understand the abstract correctly (big caveat), they aren't saying they didn't prove it. They're saying they gave two proofs, one in natural language and one in Lean, that are not equivalent to each other. I assume the main significance is that the Lean proof is not a formal verification of the natural language one and the natural language proof is not a readable explanation of the Lean one. Both of those things can be desirable, so to complete the set we'd get 4 proofs.
From computer science perspective the conclusion is obvious: untenable to have two representations without an exact translation or machine checked correspondence between then. All we have is a vibe translation using the LLM. The methodology should obviously be improved.
and clearly the computer science perspective is: get rid of the humans and express everything directly in lean so the computers can keep getting work done!
But just to clarify: is either of them actually addressing the real Navier-Stokes, or will it turn out we'll end up with two pairs of proofs about something irrelevant to the actual problem?
"3.1. When the NL paper declares stronger statements than what Lean proves
We commence with an example that complements those in §2. In the following example the AI autoformalisation results in a different Lean proof of a weaker statement."
It seems the Lean version may not be a correct statement of the Navier-Stokes problem.
They go on to say this--
Remark 3.4 (Further potential mistranslations of the Navier-Stokes proof). The above examples require careful manual checking of both the NL proof as well as the Lean proof, which is delicate and highly time consuming. Moreover, the fact that we display only two examples does not mean that these are the only cases of mistranslations.
Those are statements about auto-formalization and the NL proof. They're saying that further work needs to be done on verifying whether the Lean and NL proofs are equivalent, which is completely valid. But they're not disputing that the Lean formalization is a correct statement of the Millennium Prize problem. You can see in the link I provided (and the original pull request) that it was written by humans, reviewed by human experts, and is completely independent of the OpenAI effort.
Yes it could be that neither the NL proof nor the Lean proof correspond to the actual Navier Stokes problem. Here's a quote from the paper--
"Remark 3.4 (Further potential mistranslations of the Navier-Stokes proof). The above examples require careful manual checking of both the NL proof as well as the Lean proof, which is delicate and highly time consuming. Moreover, the fact that we display only two examples does not mean that these are the only cases of mistranslations."
The proof is about one of a multitude of possible theorems you could prove about the Navier-Stokes equation. Nobody is claiming that the Navier-Stokes equation itself is misformalized. Rather, they are claiming that what the theorem they proved says about Navier-Stokes is misformalized, in the sense that it's a different theorem than the one informally proven in the natural-language proof.
But nobody is claiming that it's a theorem about a misformalization of the Navier-Stokes equation rather than about the Navier-Stokes equation itself, and nobody is claiming that the theorem they proved in L∃∀N isn't proven. Except random ignorant commenters on Hacker News.
I don't know why this is so difficult. It seems like a lot of commenters here have the reasoning ability of GPT-3, possibly because they're extremely drunk or sleep-deprived, or are actually GPT-3.
They are claiming that the NL proof and the Lean proof are not consistent though: the NL proof is not being validated by the Lean proof, and the Lean proof is not being "explained" by the NL proof.
If I understand well, they mean that the thing they proved in lean is not Navier Stokes. And they don't make any statement about whether the natural language proof is correct or not.
Navier-Stokes is an equation, not a theorem, so there is no such thing as "a proof of Navier-Stokes". The equation is a partial-differential-equation model of viscous fluid flow. Its correctness has never been in doubt: we know it cannot possibly be an exact description of real fluid flow, that it's a pretty good approximate description, and that there exist well-behaved solutions for a number of initial conditions.
What OpenAI purports to have proven, as I understand it, is that certain initial conditions to that equation, plus "forcing" over time (which could be a literal force acting on the fluid such as stirring it with a spoon or some other extrinsic effect) only have finite (and therefore physically plausible) solutions for a finite period of time, after which singularities appear, with the velocity or pressure of some of the fluid approaching infinity as you approach the finite time limit.
I think we can be pretty confident that the L∃∀N proof is really about Navier-Stokes. The question is whether what it says about Navier-Stokes is what we think it says.
I read them as making a much weaker claim than this: not that the Lean proof isn't valid, just that it is not actually a formalization of the natural-language proof in the PDF they provided alongside it. I haven't heard any PDE people claim that the Lean proof is invalid, and I have heard things from a lot of them that imply that they think it is valid. (I'm a former research mathematician, but this is very far from my specialty, so I'm not really equipped to evaluate this claim myself.)
The claim is about the equivalence between two proofs and says nothing about the correctness of either proof. This seems to be confusing a lot of people.
>or if the initial statement in Lean is correct. I suppose that's obvious.
Is it obvious? The formal statement that Astra proved was written by a human, submitted to a DeepMind repository, and reviewed by human experts in a pull request. It's completely independent of OpenAI's effort. No expert in PDEs or formal verification has disputed it as far as I'm aware.
NS is continuous approximation to what is otherwise a discrete system. Particle collisions are discrete time events that are averaged over time. NS loses accuracy for very, very, very very low fluid densities and energies.
AI "proving" that this approximation can numerically "blow" up does not mean the approximation loses validity.
I'm dutch and not understanding his NL thesis, though i think these days its just as easy to let an ai provide a claim that is actually false, it be easier to hack lean then solve some of those solution for an AI
It can still benefit humans, but there are plenty of situations where humans prioritize guarantees of formal correctness over the ease of navigating granular details.
It seems that, despite spending decades obsessing over their various niche problems, they have little interest in a magic button that can immediately produce the answer. At best they view this button as a mild nuisance they need to contain somehow.
reply