Personally I chose not to disclose my findings in public to avoid this kind of outcome. Having a good laugh with others is not as important as keeping the result of this experiment undisturbed.
This also eliminates the chance of another round of marketing of LLM ability based on actual human contributions and I suggest everyone who is able to analyze the code to do the same - so that they cannot misattribute our ability to LLMs.
Their (public) project management is horrible, you can find fixed issues unclosed and unfixed issues closed. Not really surprising when that part of the work is completely taken over by LLM agents though.
Nope, this is demonstrably false because Rust has its own invariants around its types and the codebase is violating a lot of them.
> every potential memory issue is neatly annotated by an unsafe block
"Potential memory issue" can originate in unsafe blocks and safe code that are able to alternate the input condition of these unsafe blocks. Guess what? That still counts towards 100% in this code base, hence the abomination remark.
> refactor them out one by one
Not as easy as it sounds. They are like threads in a yarn ball, if there are one or two ends visible it's easier to sort them out. The actual situation is more like we have tens of thousands of ends (all the raw pointer code that comes with every shared object), to fix them it's basically a requirement to unwind the whole thing (rewrite all the callees and reorder the data flow as needed, redesigning all the APIs during that).
It's too early to declare it as anything remotely close to a win, optimistically saying.
For someone who is not a rust dev, would you mind giving me an example of this:
"Nope, this is demonstrably false because Rust has its own invariants around its types and the codebase is violating a lot of them."
Aw I feel quite sorry for using the word "demonstrable" while not intending to demonstrate anything, in order to avoid any contributions to the project. I apologize for that.
That being said I've found a case that doesn't do much harm even if it's fixed, and is technically not my contribution. You may take a look at that:
But code is not mathematics, so incorrect code is not worthless, it’s just worth less.
It’ll most likely still do 99% of the things people need it to do, there’ll be a an issue created for that broken 1%, and eventually it’ll be iterated upon and fixed.
Unless you’re working on a security boundary of course, there you should treat it like maths.
> So you haven't understood the distinction or you're letting your feelings for software get the best of you.
Maybe? I'm not sure.
> The proof that there are no (for rust's definition of "no") memory bugs becomes worthless
I still don't think so, the proof that there are no memory bugs is definitely not worthless, since while the whole proof itself is binary (it is either fully correct or not), its effects act more like coverage, and the more of it that happens to be partially correct, the better.
I think cve-rs is the perfect example of this. Its very existence proves that the borrow checker is already unsound, and with that, the very foundations of Rust's safety guarantees are currently flawed.
TypeScript is another good example, its type system has been repeatedly proven to be unsound, so it's also flawed.
But so what if it's flawed? Many smaller properties of these proofs still happen to hold, and they might prevent 90%+ of CVEs even as flawed as they are.
> Nope, this is demonstrably false because Rust has its own invariants around its types and the codebase is violating a lot of them.
I'm not sure I'm following what you're saying here. In the rewrite blog post it has been stated that the Rust port landed in a place where it passes 100% of the existing test suite and does so without some of the known memory leaks of the Zig version that they were aware of and didn't know how to address.
Sure, there's no guarantee that the port didn't introduce new bugs and most likely it did, but the 1.4 version has not been released yet, so I don't think a direct comparison of a long-time "stable" version and the just-landed full rewrite is meaningfully helpful.
> "Potential memory issue" can originate in unsafe blocks and safe code that are able to alternate the input condition of these unsafe blocks. Guess what? That still counts towards 100% in this code base, hence the abomination remark.
But that just means that once you get rid of the unsafe block, you're pretty much guaranteed by the Rust compiler to be memory-safe. This means that all your potential memory landmines are annotated in the code and once you disarm them all, you're memory-safe.
I don't understand how what you're saying contradicts any of my statements.
> Not as easy as it sounds. They are like threads in a yarn ball, if there are one or two ends visible it's easier to sort them out.
Still easier than the Zig version, where all the same things you've mentioned apply, but also there's no visible indication of progress and issues are unmarked and could be pretty much anywhere.
> It's too early to declare it as anything remotely close to a win, optimistically saying.
Based on the rewrite blog post it's pretty clear that the Rust version is outperforming the Zig version by pretty much every measurable metric at this point.
I think the backlash is mostly focused on the ideological opposition to fully AI-authored 1MLOC+ PR approach they took.
They can't mention what they don't know. And that's the exact reason I'm against this: doing a port without being able to tell what is correct is asking for a disaster, obviously.
I've checked their posts and blogs for a few times, and I'm quite sure they don't know any of these to write about them.
> the 1.4 version has not been released yet
Let's hope the repeated delays (it was scheduled for last week) are not for quality issues, otherwise I'll only be half correct. It appears that all the MIRI errors are not stopping them though. (They have a script to run it but most probably it just fails and they don't care. Personally I'm not executing any of bun's code to avoid summoning nasal demons, so I actually don't know for sure)
> get rid of the unsafe block
Would be possible if they are not disregarding everything else going for numbers. E.g. they are actively increasing the unsafe surface by rewriting some code in SIMD C++ (https://github.com/oven-sh/bun/pull/33032). As if no Rust solutions already exist.
> Still easier than the Zig version
Well, this is true but only realizable if they know how to deal with the new problems emerging during the process. My current conclusion is that they don't.
That's a stated goal, so nope. But releasing it before any cleanups is another story.
I'd also like to inform you that
- the current success metrics solely consist of their advertisement, my eyes looking at the code strongly suggest otherwise
- the Bun team lacks knowledge to actually make it more idiomatic: none of the Bun team has written any Rust and relying on model knowledge is already proven insufficient.
The release decision itself is presumably driven by that tbh. It's only LGTM from there when nobody knows how to review unsafe Rust.
> the Bun team lacks knowledge to actually make it more idiomatic: none of the Bun team has written any Rust
Do you know how silly this sounds? Good engineers can work in any language. At the top companies (of which Anthropic is certainly one) they don’t have “C++ engineers” and “rust engineers” they just have engineers.
Given that they are investing their time into actually learning the language before diving into this mess. No, unsafe Rust isn't supposed to be something you can handle without prior knowledge.
Jarred was even asking in public how to use traits without runtime overhead during the rewrite. And the blog post phrased it like a normal problem to encounter while doing a million lines port... That's two months after the initial rewrite so you can guess if they learned.
Reviewing is meaningless while they are still keeping the 10433 (sorry it has become 10503 since last week) unsafe blocks, most unsound and none encapsulated.
Any review would get to the simple conclusion that this should not be released before all the obvious bads are sorted out.
It isn't disabled, it has exclusions. Now that they have it, they can close the gaps. It was literally impossible to have ANY coverage before, now they are mostly, covered and have an avenue for remediation.
I don't understand why folk are having such a hard time understanding why you do large projects in multiple steps? 80/20 rule? Perfect is the enemy of good?
Was nobody here for moving billions of lines of JavaScript to Typescript? It starts with declarations, then turn on type checking gradually inside the codebase: piece by piece until done.
I like Rust and use it full-time professionally. Unsafe is not the same as unsound. Unsound means that the unsafe code is not maintaining the aliasing invariants on references required by the language, and thus undefined behavior can leak into safe code (that is, the safe code can be miscompiled).
Known unsound code should not be merged, let alone released to production. If you have good enough tests and run them under MIRI or ASAN, maybe you can get away with it for a time, but most Rust experts would not sign-off on such a project.
If somebody put a gun to my head and told me to make the best of such a codebase, I would try to figure out how to turn off the LLVM optimizations that assume the Rust references don't alias. With these optimizations this codebase is scarier than most C or C++ code.
I'm not a rust dev. But I've been once asked to take care of a TypeScript codebase and the thing was littered with so many casts to `any`, you're wondering why they bothered with TypeScript in the first place. Some people do choose tech on a vibe and not any real analysis.
My conclusion was formed in my two months long tracking of the repo activities. They have done absolutely nothing in that front. (Well, to be precise they tried to fix exactly one thing that was pointed out but that's it)
> must interface with C code that has no safe interface
Yeah so the sane first step is to create encapsulated, safe interface for them, especially in a project like this. Deno for instance have ~0.2x as many unsafes.
And mind you if you haven't read the code, the vast majority of unsafe blocks in bun are for raw pointer access to local (Rust) objects because their ownership was a mess both before and after the rewrite. Also funnily enough a lot of the access patterns are wrong (in the Rust sense), leading to hundreds of new undefined behaviors.
> be honest and reasonable
Well, well. Talking about dishonest and unreasonable behavior, why is bun releasing a new version before solving any of those glaring issues? I'd remind you the current new version is not an improvement compared to the previous one, both in terms of correctness and maintainability.
Another point of comparison is density of unsafe: the number of unsafe blocks per line of code and/or file. By this metric, Deno has a bit over half the unsafe (because the bun rewrite is significantly more lines of code).
> I'd remind you the current new version is not an improvement compared to the previous one, both in terms of correctness and maintainability.
Except, you know, multiple real companies saying that it is and using it in production. And the fact it closed all known memory leaks. And that nobody has a really pointed to a single actual issue the new version introduced after two months of endless, ceaseless bitching.
I'm also fascinated by all these people upset at Jarred for harming the readability of a codebase they've literally never cared about before it become drama. Bun has almost exclusively been maintained by Oven employees since it's inception.
> Deno for instance have ~0.2x as many unsafes.
A project written ground-up in Rust idiomatically, with a smaller surface area, still has an unsafe footprint within an order of magnitude compared to this automated rewrite from an unsafe language with different practices and known bugs? That's not exactly the slam you think it is.
> nobody has really pointed to a single actual issue
I personally know 3 categories of newly introduced issues spanning the code base with 200+ occurrences, that leads to undefined behavior and memory issues. Not doing any contributions to the code is my conscious decision, which includes not pointing to the specific problems in public.
The Zig one uses raw pointers. These are bad and get out of hand quickly but at least don't have constraints like "you must never have two &mut on the same value". You may refer to c2rust to see how "semantics preserving transformation" without new UBs looks like.
Sorry, I did. And I did an internal sharing with respect to the blog post, on topics of how to (and not to) do software migration, how (not) to deal with unknown unknowns etc. We have several ongoing software (automated) migration projects so I did rather extensive research around it.
I do regret that I've also read the code though, otherwise I'd probably live a little happier with ignorance right now. Sadly as they don't know they don't know about the issues, there's no indication of those in the two public materials they have released.
> ignoring the fact that it was an intermediate step.
They are going to release the "intermediate step" with all its issues as v1.4. I think that says a lot about which party is ignoring facts. (hint: not me, not you, not Andrew)
> The “unsafe” part is also confusing
They are right in this case. The code base contains thousands of the most unsafe unsafe blocks I've ever seen in Rust.
Yes, they have introduced (at least) several times more memory safety issues by violating the rust specific rules. Check how most of their unsafe blocks are unsound and worse, how many are straight up incorrect with a total bs // SAFETY above it, leaving no sensible usage without invoking UB.
We often ensure a project holds water against miri. This one doesn't even pass for clippy.
Well the rewrite has newly introduced many immediate and critical safety (as in Rust sense) problems. It's clear and obvious if you read the code knowing the basic rules of unsafe Rust. And don't be surprised if it doesn't appear as obvious to you or Jarred or Claude, because not knowing they don't know is the exact reason why they failed at it.
And for the emotional aspect. I'm working as an R&D engineer and one of my recently assigned experiments is to evaluate replacing workers with LLMs, quality software with slops. Every bit of exaggeration they make is making my coworkers lose jobs and I'll be no exception. The entirety of me consists of reasons going against this kind of marketing compaign.
Personally I chose not to disclose my findings in public to avoid this kind of outcome. Having a good laugh with others is not as important as keeping the result of this experiment undisturbed.
This also eliminates the chance of another round of marketing of LLM ability based on actual human contributions and I suggest everyone who is able to analyze the code to do the same - so that they cannot misattribute our ability to LLMs.