Hacker Newsnew | past | comments | ask | show | jobs | submit | mjb's commentslogin

Good article.

If you want to take the other side of my bet about the timeline, I'd be happy to send a check to a charity of your choice :)


Thank you for engaging with it sincerely! I’m not very convicted on my bet, but what do you say to pick a specific target (say SQLite?), and I claim that by July 2027 we still won’t be able to take a Rigger property (say NoRec), and prove that it holds for every single database and query in a single threaded context? (Maybe 500$ to the charity of other side’s choice, I’m not sure how this stuff works so feel free to correct me)

In my experience, LLMs are surprisingly good at it. Especially transforming user-visible promises (like "all transactions are replicated to three nodes before being acknowledged") into high-quality specifications. I'm not entirely clear on what it means, specifically, for this transformation to be sound or unsound. What definition do you have in mind?

What have you tried that isn't working?

Completeness is harder, for sure. Maybe not tractable, especially in context of the whole universe (which is a point OP makes). Applicability is pretty easy - it's a NLU problem, and we're pretty good at those so far.


I can give a real-life example. WPA2 was formally verified at the protocol level, but the formal specification -- while reviewed by multiple humans -- was flawed, which directly led to the KRACK attack.

The Needham-Schroeder Public-Key Protocol failed to include the responder's identity in the protocol, which led to a man-in-the middle attack found via model checking. This is despite numerous proofs -- some even machine-checked -- of implementations against specification.

When cryptocurrencies had their formal methods moment a few years ago, there were numerous vulnerabilities and attacks found in cryptocurrency protocols despite claims of being formally verified, and despite much of this verification being published in Rocq, Lean, etc. If the specifications are unsound or flawed, then the implementations will also be. While in these cases, it's quite clear that the folks performing the formal methods work were likely inexperienced, I don't see how an LLM will create anything more sound.

Remember that an LLM is trained on publicly available information. It's not going to make a leap above and beyond what has been published openly, and much of what has been published with respect to security is related to student projects (e.g. "Let's build a formally verified AES implementation") or one-off applications, some of which, like WPA2 KRACK, have incomplete specifications.

This is a difficult problem that experienced proof engineers get wrong, after performing rounds of threat modeling and review.


KRACK relies on nonce reuse, which as the word "nonce" implies, is something that you must not do.

Any formal specification that doesn't include the invariant about a nonce not being reused isn't worth anything at all.

So yes, incompetent people can design insecure specs that fail to state even the most basic security properties we know of.


It's not just a matter of competence: it's a matter of knowledge, experience, and tenacity. Currently, frontier model companies are trying to tackle the problem of their models writing potentially unsafe software by demonstrating that their models can build formal proofs. However, someone who lacks formal verification experience, the domain knowledge to make such software safe, or the understanding to combine the two into formal specifications will build bad proofs, because the formal specifications will be unsound. LLMs don't necessarily improve this outcome. An inexperienced engineer, with or without LLM assistance, is just as likely to not add a property to verify that nonces aren't re-used as the WiFi Alliance.

WPA2 wasn't designed by a single individual. It was designed by a committee of people including respected cryptographers and information security folks. They missed the fact that nonces could be re-used in the design, and they trusted the formal specifications that lacked a check on nonce re-use. It happens. Unfortunately, formal specifications tend to trap folks into a sense of complacency. Soundness is a hard problem, and if it's not solved, folks will make bad assumptions. "But, it's been formally verified..."

Could LLMs accelerate tactics and Curry-Howard level hole filling of specifications? Absolutely. But, they aren't likely to do much better than humans at building sound specifications. This is a difficult problem, for experienced humans with a narrow specialization in formal methods.


Do you foresee AIs (not necessarily of the LLM kind) getting smarter enough to build proper proofs?

It's difficult to speculate about the future. We don't yet have the right kind of hardware -- whether it be performance, scale, etc -- to simulate the kind of synthetic intelligence necessary to outpace humans at reasoning.

It's not a matter of constructing proofs. This can be done via brute force with appropriately managed search trees. LLMs actually are not much better at this than other ML or search techniques; it's just that in practice, that's not what proof engineers have optimized for.

Building a sound specification, that as erichocean pointed out, would include measures to prevent nonce re-use, is a much more subtle and difficult thing. You can prove that software is correct with respect to a specification, but there is nothing beyond proving internal consistency, to verify that the specification itself is sound, complete, or appropriate. This is an open-ended problem, and one that humans have difficulty with. I don't think that any synthetic intelligence is likely to do better, because it can only do as well as the vetting of information provided to it, and it is very likely that this information will be flawed and incomplete. LLMs try to fill in the blanks with plausible things, based on their training. Even humans make assumptions. These assumptions are sometimes wrong.

Regarding proof automation, this is actually a double-edged sword. I can build systems today that will perform various optimization techniques to try to get further into a proof via tactic search, and will eventually converge on a solution. However, the tactics will be a mess, much like the 1600 pages of tactics in the OpenAI NS proof. A good proof engineer builds specifications and proofs that other humans can read and understand. Even the tactics are organized around what can be explained, so that these can be hand checked. If you're building proofs of software, you want auditors to be able to look at these as well as the specification to see if you are missing something, such as a nonce re-use policy.


The flaw exploited by KRACK was not in the proof. I thought that was obvious.

It's that a key property, that a nonce (a number only used once) was never reused, was never stated. So that property was not proven, and in fact, was false.

It takes one line of specification to state the nonce invariant. It is beyond incompetent to leave it unstated.

Once stated, any proof would have to also satisfy that invariant (and the existing proofs would, of course, fail since the current design does not prevent nonce reuse).


In this case a reasonable conclusion would be that there are no sufficiently competent people and we should act on that basis. Or, at least, that we have no reliable way of marrying sufficient competence with the right problems.

> there are no sufficiently competent people

The common countermeasure to omissions and mistakes is a social measure: we bring talented people together to work as a team.

Even doing so, failures happen. Wise people know the value of testing.


Well yes, but this case is specifically about a time when the right thing wasn't tested for, because the team of wise, talented people attached missed it.

I regret to inform you that the word nonce implies something else entirely in the UK

I disambiguate by saying "cryptographic nonce" at work.

I think OP is referring to more of a fundamental problem that a spec has to be written down, and if it's not then AI is just guessing. Often it makes very good guesses, but it can fundamentally never solve the problem of producing a complete specification when the user hasn't got a "complete" picture in their mind.

In my own practical experience this is now what software development is about. If you can specify what you want, AI can implement it. But specifying it is actually very time consuming and hard.


In a sense every prompt you send to an LLM is a specification. And its tedious to write all those prompts, I feel. You have to be very precise, and also write it all into the prompt. And even then answers I get from the LLM are often lacking so I have to prompt again, and then often the LLM replies that it now agrees its previous answer indeed was wrong or inapplicable.

I agree, but that's the job now.

One obvious issue is that an LLM can claim it has formally verified your spec, only for careful inspection to reveal that the spec was subtly changed in some way during translation. Meanwhile the LLM continues to claim it implemented the spec exactly, then pulls a "you're absolutely right and here's why" when you point out the issue.

I think everyone who has used LLMs for coding will have run into this at some point, so it's not very convincing that LLMs are good at this sort of thing.

We have the case of the paper version of OpenAI's Navier-Stokes work turning out to be describing something slightly different from the Lean formalization, as a more public example.


> For any first-order universally quantified property of a program we can write as a property-based test, I think we will be able to generate proofs at the binary level, against architecture models that have themselves been validated by millions of hours of testing. The challenge of extending this to multi-threaded programs, distributed systems, [..] will be years more of work we must conduct across thousands of verification engineers and researchers.

I expect the distributed and concurrent version of this will come more quickly. In fact, I would be willing to bet that by EOY 2030 the bar mentioned here of single threaded software will be met for multi-threaded and distributed systems under non-adversarial failure modes. This is significantly more aggressive than most in the field (and I'm field-adjacent at best), but given the rate of progress I think it's actually a rather conservative bet (the really aggressive bet would be "July 2027").

> The more "bouba" a piece of software or a given layer is, the harder it is to talk about what makes it correct. We can always talk about some correct attributes, but to talk about a uniform notion of correctness is not possible in my opinion.

I largely agree with this, although I do think we're going to see more progress than expected in quantifying (and therefore verifying) usability and "look and feel" aspects of software. The risk is that this happens in a "narrow" way that forces all 'correct' software to look and feel the same, because that sameness correlates with the learned definition of 'correct'.

> In solving these challenges, I think random testing and random exploration in general will prove to be invaluable tools. A test execution is a measurement, so it produces information about the real underlying system that we can leverage to falsify, harden and validate the models we use for verification. The verifiability of testing will continue to be a huge driving force of its adoption as well as a gateway into the rest of formal methods, and maybe it will even help us trust when we hear software is "formally verified" instead of wondering what was left out or when someone will find a bug in there.

Strongly agree with this.


There's a pretty interesting product i've wanted to try that combines fuzzing with a deterministic simulation testing: https://antithesis.com/

unfortunately, they don't have any tier suitable for solo devs


> under non-adversarial failure modes

There's no distinction to be made if the software is verified. If it fails under adversarial situations, it has failed verification.


I'm nobody when it comes to formal verification, but I believe there are a lot of simplifications to be made if you have additional assumptions about your system, e.g. information can't be extracted through side channels, or that proxy you use is airtight, or that operating system your systems is designed to run on schedules the tasks correctly. All of these things might not be true, but assuming they're not true is a PITA (and only present under adversarial situations) and leads to a question why don't we completely verify what's underneath first, which is reasonable but not cost-effective.

That is about the scope of what is verified. You often choose to only verify one component, but usually you want to build on a foundation of verified components.

The bigger problem is that you usually want to verify a model of the system (higher level, a protocol not an implementation), which has better tooling and solving properties, and then you have a problem of proving correspondence between the model and the implementation. LLM driven proofs could be game changing for that last stage, and better at handling non-ideal machines.


Love the idea of quantifying/verifying usability - maybe it isn't one thing but it's like an envelope, a 2d (or many-d) region, since we recoil at there being just one correct solution to what feels subjective and range-y

And yet, the specification of almost every program in existence was communicated in natural language. At least between the customer and developer.

This argument makes the implicit assumption that humans are uniquely able to convert between this "customer wants a program that solves their problem" level and compileable code. That was a reasonable thing to believe in 1978, but is not a reasonable thing to believe in 2026.


Once an employer asked me to automatize his work, so the task that he did in weeks was done in seconds. I made a gui that showed every step he did manually, so he could check and understand every step. And yet he became confused and lost track of what the program was doing. He frequently asked me the very concepts he taught me so I could do the program. I just implemented exactly his specification. I didn't created any new concept, but as soon as the task became automatized, he stopped understanding.


> This is based on one of the smaller Qwen models, just like Cloudflare's Clef, Strands decider

As of this afternoon, we also have Gemma4-based variants of strands-decider at 2B, 4B, 12B, and 26B: https://huggingface.co/StrandsAgents

I don't know if Fabio (who's leading work on this model line) would agree, but if I was going to start over I'd probably pick Gemma4 as the base rather than Qwen3.5. But it's not surprising to see a lot of Qwen competition, and I totally agree it's nice to see the work open.


I been using gemma 26b moe model as the system one variant for my own uses and its great and far smarter then any alternatives at around 200ms, bigger is better for these things as long as you dont need lower latencies, only caviat is calibration, if you need calibration better use JEV.


Yep, Fabio here, and I agree!


Oh, huh, that's an error in the image I didn't notice! Those comparisons are to a model called 'decider-2B', which isn't ours.

These benchmarks cover strands-decider v19. v21 is better calibrated, and we've got some new ones coming this week that move us further towards the bottom right.


One of the authors here.

I appreciate what the JevBench folks are doing, but it's true the JevBench (like all benchmarks) isn't representative of every real-world workloads. I particularly don't love how JevBench handles confidence, and rewards being highly confident in certain cases. Benchmarking is hard, and JevBench is probably more representation of its target workloads than TPC-C is :)

As for actual intelligence, there's only so much you can expect from 2B without reasoning. One of the design challenges in training this model was to avoid forgetting too much, and the KL-to-frozen-base step partially exists for that purpose. Even then, Qwen3.5 2B base has fairly limited single-pass reasoning ability (which shows up for us in the performance on the JevBench hard set).


It's very doable, but we haven't done it yet (although some great community folks did an ONNX version of v21).

I haven't looked in depth, but it should be doable without modifications to llama.cpp. You can't just convert the LoRA, through - there's a whole pointer head and some custom layers that need to be correctly handled (in fact, the LoRA mostly exists to get the base model to behave the way the pointer head needs).


One of the authors here. Thanks!

On humans-for-humans, blog posts with my name on aren't written by AI (https://brooker.co.za/blog/2026/06/18/my-blog-and-ai.html) either for my personal blog or at work. I'm a heavy AI user, but this is something I think is best left to humans.

There are a ton a ways to use models like this. Model routing is a popular emerging one. The one that really interests me is hybrid agentic workflows - filling the gap between deterministic workflows (e.g. AWS StepFunctions) and fully agentic workflows that use frontier intelligence.

We're releasing more of that functionality in strands, and I expect an explosion of innovation in that area.


I wrote a slightly deeper blog post here: https://brooker.co.za/blog/2026/09/28/engineering-system-one...

On multiple tokens, we put each option (however many tokens it is) onto a line, and then read the hidden state at the end of that line to use at the option state. This is done option-by-option.

The hidden state at the end of the question is the query `q` (again, not sensitive to how many tokens the question is), each options end-of-line state is the key `k`, and the per-option logit is calculated as `q.k / sqrt(256)`.


Appreciate the transparent process! Is there a reason why you chose to process the entire list of options as one input to the LLM torso instead of splitting them and processing options separately? From what i can tell from my (limited) experiments, there is a lot of variation caused by simply reordering options, plus a heavy bias toward the first option listed [0], and my guess is that this is caused by hidden state of each option "leaking" into subsequent options

[0] https://news.ycombinator.com/item?id=49987076


No particularly principled reason, no. It's one of the (many) design variants we haven't had time to experiment with yet.


Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: