Agree! 'Problems' are getting solved and this needs to be celebrated. Wondering how this will discourage mathematicians at all, since now they have another tool to accelerate their research. Nothing is stopping them from using 'new technologies' or sticking a gun to their head to use the 'new technologies' either.
If you consider this event in isolation it is cause for celebration. But the controversy around this isn't so much about how the proof was obtained but what this means for the practice of mathematics going forward. It seems we can probably expect more and more results of this nature being dumped into the community. It's happened before that one person, Bill Thurston, was so successful in his field, proving theorem after theorem, that he inadvertently killed his field. People hesitated to enter his field, knowing that they could be scooped at any moment. It took years before his field recovered - and I think his famous essay was written in response to this.
In December 2024 o3 scored 87.5% on ARC-AGI-1 and cost $4560 per task.
DeepSeek V4 Flash 0731 scores 89% and costs $0.02 per task.
If we apply the same factor to the guesstimated API price of $20M for this problem, we arrive at $57.
Real cost is a fraction of the API price. Although the internal model might have a higher API price than the ~$19.5M I estimated based on Astra's pricing.
Yes, but a big part of the problem right now is that two big labs have monopoly on the resources and they for sure are not working for the benefit of mankind.
World War III was the last of Earth's three world wars, lasting from approximately 2026 to 2053. The conflict involved nuclear cataclysm as well as genocide and eco-terrorism. The post-atomic horror in the aftermath persisted as late as 2079.
The war was preceded by the Eugenics Wars and the Second Civil War, all of which were sometimes regarded as parts of a single escalating conflict. It resulted in the deaths of some 30% of the Human population, at least six hundred million people, and the extinction of six hundred thousand species of animals and plants. By the end, most of the major cities had been destroyed and there were few governments left.
Early lesson I learned from AI engineering was - there is no substitute to giving a groomed epic to an agent. Instead of simply saying 'implement themes in my product' you need to be specific, in fact more specific than usual. You need to say exactly what is in scope and what's not, even down to a buttons, events and layouts.
You can groom the epic with the help of AI, but final review must be done by someone who can take ownership of the specs and hence is responsible if something has fallen through the cracks. AI's response will be limited by the output tokens of that specific agent, and there will no repercussions for AI even if it accepts its mistakes.
Yeah, I think as a general direction we need to allow a useful collaboration with the agents on the issue board.
This is btw why Epiq was developed, to keep the board as code, git-backed, distributed (via an event log mechanism), and with the ability to replay the board, to see what agents actually did:
I hope @jolaflow can chime in here eventually, but from a brief look, my impression, besides the fact that Epiq is based on git as opposed to beads where it is optional, is that Epiq seems to be much more optimized for interactive collaboration between the user and the agents.
The graph visualization in beads surely is a neat thing for showing things, but the replay feature in Epiq should provide a similar understanding of what happened.
But again, it seems to me Epiq is the tool that better allow the user to jump right in and collaborate with the agents on the board.
(Again, this is from a brief look, so I could be missing things).
Sorry, I might not have been clear, but the tool and blog is by @jolaflow (my brother) and I'm so far mostly involved as a reviewer and early adopter, but glad you like it. I might be biased, but I think the tool provides something really useful for agentic coding and hoping it can grow into a thriving open source project.
> Early lesson I learned from AI engineering was - there is no substitute to giving a groomed epic to an agent. Instead of simply saying 'implement themes in my product' you need to be specific, in fact more specific than usual.
I think this lesson is getting partially outdated. Yes, you need to be specific about what you want, and with earlier LLMs, you need to had both domain knowledge and some general software development experience to front-load various big and small choices about design, architecture and operational reality - what libraries to use, how system components communicate, how you handle auth and store secrets, etc. Otherwise the LLM would pull some random mix of ideas from its latent space, and give you something that's broken in really stupid ways.
Nowadays, it doesn't feel like that to me, not anymore. I still need some understanding to verify the proposals, but I found the last ~6 months of SOTA models to make good choices. Like, just yesterday I asked Claude to design me some simple service, and focused on explaining it the domain parts (nature of systems I want to integrate together, the purpose of that, and the user's priorities and use cases), and the design I got back had specific suggestions around security, authentication, deployment, failover, integration, behavioral impedance-matching between integrated systems, and more, that I all recognized as based on solid software engineering and ops practices, but deviating from it explicitly in every place where it would be wrong for this specific project. The model considered way more corner cases than I did, and I'm actually really impressed by it.
But then, I find greenfield development is easy with LLMs. Modifying existing systems, especially legacy ones, is where I need to babysit and micromanage models - because any misunderstanding or inaccuracy, which often comes from stale documentation or naming mistakes, tends to get amplified and confuse the agents. No matter how precisely you specify your epic, if the model will find something that contradicts your knowledge/intent, there are good chances it'll get confused and make subtle errors, and you won't realize until much later.
The way I see it: models are highly biased to treat everything they read as "ground truth", all of equal importance. There's no nuanced notion that some information may be stale, that there's a temporal and causal order to sources, and that some information may just be wrong.
And this compounds when you let your LLM write code and documentation over time.
>> No matter how precisely you specify your epic, if the model will find something that contradicts your knowledge/intent, there are good chances it'll get confused and make subtle errors, and you won't realize until much later.
True! hence the need for someone to review the final spec output and own it as their own output.
I have also found LLM to be better at debugging and solving 'a' specific problem, which I believe is due to output's surface area to be reviewed is lesser in comparison.
But isn’t “grooming” the hardest part? To know the details means to know the codebases behind. So you do discovery, read code, ask people, check db tables, and then once you figure it out, you write a doc with the proposed solution. It gets reviewed and then you code it.
I definitely need AI help for the discovery part… so it always starts with a simple “I need to do X”
Genuinely curious, what is the benefit of doing this over just programming it yourself? It sounds to me like this is just an extra step that will grant you less control?
When programming a feature, you are simultaneously doing at least four things:
1. Implementing.
2. Building highly detailed mental models.
3. Learning and expanding your skillset.
4. Quality control and scope limiting.
And this process can be iterative and dynamic. Writing massive, super detailed specs that you then hand off to a undeterministic model feels like doing step one and three, while skipping two and four, which you then have to do after. What is the benefit? The speed up, in my opinion, comes if you skip step two and four, but then your product WILL be worse. Feels like I am going crazy?
Programming was never the bottleneck for software dev?
If you define what you want in typescript by updating existing typescript wordage, you get what you want exactly as specified, with minimal token cost and no ip theft.
Not sure about that. Writing detailed specs you deal just with the inherent domain complexity. Writing code you also get accidental complexity that has nothing to do with the business, whuch not only piles on top, but tend to interact vexingly with the detais of the domain.
Where you would have a point is if you'd say we have excellent tooling for wrangling code, but less tooling and tradition to write and manage specs.
I have realized that it's like giving a task to a brilliant coder who has just joined the org and is more excited and eager than usual. Hence the responsibility falls squarely on you to set scope constraints while ensuring only to-the-point features are developed.
> I honestly feel like basically nobody knows anything about these models, it's all just vibes
This, too. Since only providers know what they actually serve, what they change and what limits they impose.
There are some visible degradations though. E.g. Claude-ish.
As for a personal anecdote: around February I created a rather complex quiz web app for myself and friends with multiple question types, sync between screens, multiple media upload types, multiple scoring and timing types, MC inetrface etc. etc. etc. It took me a week or so in the evenings with rather vague prompts to make it.
Now Claude (and Codex) cannot reliably build a much simpler web app even with precise instructions while also maintaining the visual consistency.
But I will agree with you, it's a feeling, not a precise measurement.
I am guessing (guessing) that "great coding model" and "great model for the widest use" (agentic, long-running, unsupervised, knowledge-worker, computer operator) are competing, conflicting endgames.-
Add to that:
- Of course, "labs" (quotes) are incentivized to throw coders under the bus and aim for the biggest possible market.-
- "Sharp, focused, brief, elegant, precise" editing, as would benefit the coding use-case, is actually token-saving, ergo, undesirable.-
The only thing that can stop this, would be the quality and functionality of the codeslop generated by these models to became so low that it actually interferes with the (alledged) recursive self-improvement of models (ie. models start to perform worse/degrade).-
Until such a time, we serfs, will eat what's on our plate, pay for it, and continue to kneel before the machine god overlords. And be glad for it.-
Probably because you were impressed by whatever it produced to fill in the gaps at the time. But if you expect the models to read your mind you're going to have a bad time.
If mathematician was already using OpenAI for research purpose and making progress due to inputs from OpenAI's responses, then I wouldn't put it beyond OpenAI's reach to generate different relevant prompts to make progress by itself. Afterall, Model can keep at it for whatever timeline and keep pursuing all possible combinations it can think try.
Harness definitely matters more from safety and reliability point of view, but saying that it matters more than Model itself is a slight exaggeration. In the sense that this 'headline' can lead people to believe that all models are equal and Harness can make up for lack of in-built features of a model.
Internal model must have native features like mixture of experts, memory features, etc. for the harness to use.
Actually, we do need evidence for premises which are not definitions or basic assumptions (axioms). Graviton is still a hypothetical particle and falls under neither of two categories - Definitions or axioms.
The statement "if gravity is quantum, this was an interaction with a graviton" is true even if it ultimately turns out that gravity is not quantum. The statement is about the consequences of a premise, and is not falsified if that premise is false.
The only way this statement could be false would be to have quantum gravity but no particle that mediates this interaction - which doesn't seem plausible almost by definition.
That's quite an exotic claim! If Sound and Temperature can emerge without a sound particle or temperature particle, why is it not plausible for gravity to exists without graviton?
Though, I get it that mainstream view from physicists is Graviton is the most 'likely' cause, if the gravity is proven to be quantized. But even they would have the humility to accept that this is a theory yet to be proven and observed!
A: If gravity is quantum, then (B) there must be some particle-like think that we call a graviton that mediates the interactions, and then seeing a rubidium atom fall must have been an interaction with this particle.
not A: If gravity is not quantum, then they are making no claims about a graviton.
If you think gravity is not quantum, then you go to the "not A" branch, and they make no claims about that branch - so there is no contradiction with their IF.
However, for me to claim that quantum Gravity is only emergent without a particle like graviton's mediation, I would need to present evidence. Just like this claim - quantum gravity is only possible from gravitons.
Ohh, yes, I had not at all understood that this was what you were referring to, especially as I had already suggested this is a real, though to my mind implausible, possibility.
Now, as to how plausible this is, we'll actually get into semantics. To me, saying that "gravity is quantum" would mean that it is a real force/interaction, and that this force obeys the Heisenberg uncertainty principle. So, I wouldn't say "gravity is quantum" if it turns out that the Entropic gravity theory is true, as in that theory gravity is only an apparent interaction. However, I admit that this may only be my perception of what these terms mean together, and that others may use "gravity is quantum" to refer to any theory that explains the macroscopic effects we call gravity in a way that matches quantum theories.
So, basically Gravity is quantum if Gravity is fundamental (not emergent)?
If I am understanding your counterpoint correctly, then this argument can only be concluded satisfactorily if either Graviton is observed (proving is existence) or something more fundamental and deeper is observed (proving that quantum gravity can emerge without graviton).
Military Ops, Armed Conflict, Defensive strikes, Global war on terror, counter terrorism ops to name a few. These are the labels given to recent conflicts like Afghanistan, Iraq and Vietnam wars.
Most solutioning is art, given that you should have more than one way to reach the target state and target state itself is negotiable and non-concrete. When there are multiple options you are forced to analyze between them, sometimes you have a preferred option if the analysis has been done before. Your choices while solutioning will reflect your experience and how you think. In short, it definitely reflects 'you'.
You make similar kind of choices while programming as well. Hence, programming is definitely an art since there are many ways something can be implemented, almost to the level that you can recognize who has written the code by seeing the code itself. Most such programmers don't even have the visibility to higher level goals of the org they are working in, I used to be in this group earlier.
I remember it used to be a 'problem' for a lot of senior developers, who used to ask for consistency in how code is written, like spacing between keywords, indentation, start/end of parenthesis, naming convention, etc. Wondering if they'll settle for more 'Soul' in the code now :). Or maybe, they have permanently shifted to the other groups now - where only the output matters.
> Most solutioning is art, given that you should have more than one way to reach the target state and target state itself is negotiable and non-concrete. When there are multiple options you are forced to analyze between them, sometimes you have a preferred option if the analysis has been done before. Your choices while solutioning will reflect your experience and how you think. In short, it definitely reflects 'you'.
This is an very broad definition of 'art'. By this metric, I'm pretty sure me choosing what to have for lunch is art.
I think art's definition would be even broader. Anything that involves your imagination, skill, ideas and analysis (applying yourself to the choices in front of you) would be art. Just googled and verified this, btw.
If choosing what to have for lunch is making you involve your imagination and analyse choices, then I guess it would fall in the category of art.
And due to restraints of time, budget and prior experiences, you might go for default choices anyways.
I very much doubt that they listen to countries in tier 1 and 2 also. If not for the fear of US's relatively stronger regulations, don't think Meta would even pay attention to any complaints in US either.
This would not change with the change in leadership in Meta simply because 18Bn is fairly affordable for Meta's scale, but they should be worried about setting the legal precedent (though they denied any wrong doing).
Other countries may not have the same ability to impose penalties owing to their weaker laws also.
Another AI-as-amplifier vs AI-as-substitute article.
Wondering what happens when tomorrow AI accepts and learns from the feedback and gets trained on all the failed initiative as well. It wouldn't be so hard if failed (something which wasn't right at all or hasn't got the traction) projects and ideas are all listed somewhere for an LLM to scan through. LLM may finally figure out how to add personal scars, hard decisions and personal insights which can be personalized further.
Also I don't think it might be so undesirable for an org, if there is a system which observes and present hard facts based on last quarter or year JIRA (pi planning and sprint planning) and commit histories.
reply