Yes, if you mostly have no extra cores already or if you have very wide unrelated deps, this mostly does nothing. Typically, this is not the case though so there is some good perf gains.
I wanted everything in one repo and this worked well enough. For presenting to rust compiler team I forked the repos in question and made branches for review with the patches.
(Copying and pasting from what I posted in rust zulip)
There is an .early-rmeta file that is generated that contains the result of type checking the outer part of the functions (arguments, returns) that is then passed down to all the other dependencies that can use it and get a "headstart" on their work. -Zearly-metadata is the flag for Rustc here to turn that on. It also needed to learn how to swap the real metadata in for the early stuff that it was previously using, once the real stuff showed up.
Rustc needs to be able to pause and resume at certain points and Cargo needs to be know how to look for and react to that, which is what -Zheadstart does for Cargo. So there's some changes job_queue that happen to there. I haven't checked this directly yet, but my understanding is that it doesn't kill the process, it just has it sit and wait, which normally would occupy a slot, but doesn't any longer.
---
So not a cache, just being able to pause and start work and better utilize the available slots.
Sounds like if this was incorporated you could include metadata files in crates and even avoid that analysis cost being repeated? Something like typescripts.d.ts files?
It’s a neat idea. My impression is that it would go against the grain of the compiler currently. The rmeta file and lots of the other rust outputs aren’t meant to be that stable that it could be portable like that. I’ll certainly mess around with this some and see if anything comes of it though!
Talking with them presently! This specific set of changes won't get in (LLM written, little to no thinking through of the broader design), but I am hoping something like it shows up sometime.
Never thought of it like this, that synthetic code doesn't need to get into a codebase for it to benefit from the discussion around what was encoded with LLM.
Sucks when four out of twelve are flawed and you rule out the top 2 best solutions and 5 useful test cases you could learn from by just spray and praying... But at least it was fast!?
Yeah, that's not the way... It's synthesizing a single hypothesis at a time with attention to the emerging architecture, then decide to reimolement manually or reapply agents to refactor the code until it's reasonable.
The people I've seen do things like this blew through good ideas because the implementations were flawed and they couldn't be bothered to you know read the code because people ... Are told not to do that anymore. In some situations they are not even allowed to read or discuss code at work anymore. It's ridiculous...
Yes it's a pretty well thought out policy. It feels bad for someone to put in a month of work and have their code ultimately rejected, but if an llm does something sloppy with some light guidance in a few days, then great, experiment with some weird stuff and see what happens, then once you know it probably should work, go in for that month of work and make it happen.
That's a good point. I've gotten a few PRs to a project I work on, where the general idea is good, bit the actual code has serious issues, so I end up just implementing the idea myself in a better way.
It's a decent idea until 12000 bots start spamming a repo with janky code and poorly founded really complex suggestions for cs students to spray and pray resume padding.
Funny to see this now. I’ve got a private branch I am in the process of shaping up this weekend to show the compiler team. For deep nested projects like rust analyzer, if you emit the meta data about function types earlier for downstream slots to use, before successful type checking, you can start other crates earlier and use all the slots you have instead of sitting around waiting for the full type checking of the bodies (which other crates largely don’t care about). Something like 40% wall time speed up, maybe 10% to 15% if you have the parallel frontend on.
Nice, excited to see that. This is basically deeper pipelining, IIUC. Nick (the author of the blog) implemented the current version of this "emit metadata sooner," optimization and already experimented with pushing it even sooner a while back.
We have discussed this briefly on last All Hands, the problem will probably be dealing with compiler errors, when they happen in builds that already emitted metadata. I believe this was the reason why it wasn't merged in the first place.
Yes I think I read a post of his from like 2019 were something was tried like this before. My guess is that this will be an interesting and hopefully informative PR for you all to see, but will likely not be merged as is. I had to add the ability for rustc to pause after early meta data emission and that was hacked together I feel, not well designed per se.
I view this as speculative execution/compilation and just throw out any errors from a crate that depended on another crate that ultimately errors out. It has the same outcome (same errors are printed in both cases) you just have a chance of having totally wasted some cpu time (that was otherwise just sitting around though).
Sounds lovely, give it a try on the codex-rs which seems to be getting close to using 2K crates in their repository any time now. Takes like 30 minutes to compile on a my beefy machine and it's a goddamn TUI, not sure what's going on, and not too interested in diving into that beast either.
It has 1.8 million lines of rust. That compilation time seems close to what I'd expect at that size. If you want something faster you should focus on distributed build caching rather than speeding up the compiler. Also note that crates are effectively equivalent to a single file in c++ as it's the compilation unit. This is similar to compiling 2k files that average 1k loc each in c++. I'm sure it's much worse when you account for macro expansion and generics.
As they say, it's a damn TUI for a chat bot. 1.8 MLoC is like 50x what you'd expect from something like that. If you want to speed up compilation, reducing that absolutely insane bloat would be a good start
> If you want something faster you should focus on distributed build caching rather than speeding up the compiler
... But the whole point is that that project is big, unwieldy and has lots of crates, both their own and 3rd party. Slapping a cache in front of it kind of defeats the purpose here, to see how parent's thing would speed things up for really large projects.
Sure, that compiles in a few minutes for me too, give it a try with a release build and compiling the full workspace though, since the point here is to stress the compiler :)
If there's a project that takes 45 minutes to produce a debug build from scratch, that's not a Rust problem, that's a them problem.
Let's compare. I just checked out ripgrep and built it and all its dependencies from scratch. Total time to produce a debug build: 6.92 seconds. This is inside a 16GB VM on a random Linux laptop (on battery); no beefy hardware here.
Let's try something beefier. I just checked out uutils (an implementation of coreutils) and built it from scratch. Total time to produce debug builds, for eighty binaries and all their dependencies: 30.26 seconds.
Let's try something beefier. I just checked out Servo, an entire browser engine, and built it and all its dependencies from scratch. Total time to build and link its one-thousand one-hundred and ten compilation units (along with its infamously huge and serial servo-script bottleneck) into a binary: 4 minutes and 38 seconds.
If Codex takes 45 minutes to build from scratch, that's an indictment of OpenAI and their development processes.
Arguably it is a rust problem. Rust can be improved to reduce the need to do as much manual maintenance of crates dynamics to allow for faster build times. The "you're holding it wrong" answer is not a good perspective to apply and I am surprised to see a prominent member of the rust community using it.
No, one can write pathological code in any language. I welcome efforts to improve the performance of the Rust compiler, but nobody is served by optimizing for degenerate use cases to the detriment of legitimate use cases. It is, in fact, possible to hold it wrong; this isn't blithe dismissiveness, but rather an accurate statement of reality, and pretending otherwise doesn't lead to a better language.
I feel like there is an assumption here that solving the problems described will result in a worse outcome for the average rust user, but I'm not sure that's the case. Perhaps the existing way of compilation makes sense for release mode, but a different tact could be taken for compilation modes which are optimized for developer productivity.
I agree with the sentiment that you can't solve for everyone, but if a problem is prevelant in your userbase, it's worth optimizing for solving it. Not every tool will be used the way it was designed to be used.
I've abandoned zig for rust. I expected to be annoyed by rust build times, but so far it's been quite good for my needs. To the contrary it has not been any impediment even when prost etc all drag in many deps from a cold build.
The zig compiler apart from a couple of cross build or linker issues is darn good. Switch to rust is a function of library support, private struct fields. Rust itself is better organized and far better documented compared to zig.
However, rust has got to get their allocators for container support straightened out soon. I do SAAS for fintech and low level packet transports. Both benefit greatly with allocators custom designed for job at hand.
The allocator API has been stabilized, to be released in Rust 1.100 in November. (Though only Vec and Box will support custom allocators immediately; the rest of the standard library collections are coming later.)
> Incremental compilation is not cleaned up. Older deps pile up and don’t get cleaned.
This is one of those problems that’s both huge/important, and probably impossible to solve. It bites me all the time, I have a measly 1TB nvme disk and I basically have to wipe my target directory and docker build cache (I have a shared cargo cache volume across builds) every other day. I’ve had this workstation for 2.5 years now and I’m honestly worried about nand endurance coming to bite me soon.
How would one even solve this? When do you know a cache entry won’t be used any more? Trace back provenance of artifacts to their sources, and if the mtimes have been bumped just delete the artifacts? It’s not like resetting source files to older mtimes is a good idea anyway…
I disagree it’s impossible to solve. Huge C++ projects don’t have continuously growing build directories and can still compile incrementally correctly.
That’s because they overwrite the old artifacts with new ones every time, which has its own problems: you could end up reusing stale artifacts if your compiler version changes, or if different compiler args are used for the same artifact, and a bunch of other stuff. Rust computes a digest of all the relevant build environment settings that may need a different artifact to be built, and uses that digest in the file name. That means you basically never have issues like “oh crap the incremental build broke, do a clean build instead”. (Amongst a whole lot of other benefits I’m sure.)
But this technique errs on the side of not using invalid artifacts, at the cost of not having a good system for expiring them. To maintain the goal of never having to worry about when to require a clean rebuild, solving cleaning of stale artifacts becomes a much harder problem.
> It’s insane. 45 minutes to build debug from scratch. Linking is super slow even in debug and that’s without LTO.
How much memory do you have, and do you have swap turned on?
I have 32GB of RAM, and it barely manages to build release if I shut down everything else on the system that uses RAM. 64GB would be better. If you have 32GB or less and you have swap on, the performance might be mostly because you're spending a lot of time swapping. And that's for release; debug on a repo that size may spend a lot of time and memory linking debug info.
I will email you when I have something up for sure! First cargo check has it at about 28% reduced on codex rs, from 383 seconds down to 277 seconds on a 16 core machine. No -Zthreads yet though, and that typically has a similar effect and reduces the speed up the early meta data gives.
Depends on what you are editing I would say, if it's something deeper in the stack then it should help. The last numbers I have is that this is a 6-8% improvement for incremental editing of the code for cargo itself, but I'm having the benchmark be rerun now, as I've landed a fair amount of changes since then.
Mother of Learning is pretty good, similar vein, does some interesting things. The Wandering Inn is also pretty good, I'd say more challenging than Dungeon Crawler Carl while still being entertaining.
I don't see why mathematicians should be protected from AI anymore than any other profession. It's either everybody or nobody, not fair on the face of it otherwise.
Mathematician was never really a "profession" like the others. It doesn't pay well and is largely confined to academia. If you're really good and want to get paid, you don't do the kinds of problems AI have been taking a crack at. You go to a quant firm or some tech company where this kinda math actually matters once in a blue moon
TFA isn't asking for mathematicians to be protected from AI. It's asking AI labs to hold themselves to the standards of the mathematical community:
- releasing papers using the normal process to allow peer review
- giving talks etc to disseminate knowledge so humans understand the result
- writing papers in a way (standard terminology etc) that allows mathematicians to digest the result (some AI math papers comprise a huge verbose load of non-standard terminology and waffle and then a massive lean proof. This is very hard for humans to actually understand, and means it's hard for others to take the work forward.)
- giving appropriate credit to results that are used to derive the work
It includes some specific recommendations for situations where the person prompting the model is not in a position to understand the output, and frankly these are really welcome given situations like the recent case at Anthropic where a non-mathematician at Anthropic prompted claude to make a significant improvement to the bounds of a problem related to the Riemann Zeta function[1] which led to widespread misreporting and claims (not by Anthropic themselves notably) that the Riemann hypothesis itself had been proved, which is emphatically not the case.
Research mathematics is fundamentally a collaborative activity and the way in which some of these results are released is done to maximise PR but means a ton of the mathematical value is left on the table.
[1] https://www.anthropic.com/research/riemann-zeta. As I understand it, the Riemann Hypothesis says that all non-trivial zeroes of the zeta function lie on a line called the critical line. Two centuries of previous work had established that at least something like 40.9% of the zeroes lie on the line and noone has ever found a non-trivial zero that does not lie on that line. Claude (with prompting from a non-mathematician to "try harder" etc) improved this bound massively to 67%. Now a lot of people said things like "OK so all we've got to do is to improve that to 100% and we've proved the RH", which is definitely not true unfortunately, because you can say that in the limit the proportion of the zeroes on the line is 100% and still have infinitely many which are not.
we do not endorse this practice, and we ask them to stop testing advanced mathematical problems on proprietary models
From point 2 under section "1. Background":
AI labs should provide significant support, including funding, to help develop this understanding
Point 3 under the same section:
The development of human understanding must remain organic and community led. It should not be directed by AI labs, even when the labs have produced the results.
If, as you say, they are "asking AI labs to hold themselves to the standards of the mathematical community", I must conclude that the mathematical community
1. Ideally wants a monopoly on mathematics research.
2. Demands money from those who dare violate their ideal monopoly.
3. Insists that the ideal monopoly remain in charge.
Your monopoly theory lacks a credible motive and also ignores the fact that your quote specifically targets the frontier labs, not LLMs or computational tools in general. The group (https://agmai.org/) comprises of world-renowned mathematicians, including several Fields medalists, who have nothing left to prove. If they felt like it they could quit mathematics today and take private sector jobs paying far more than their professor salaries.
The basic purpose of mathematics has always been human understanding (see for example
https://mathoverflow.net/a/44213), and the working group's various recommendations simply aim to ensure that computer-generated mathematical activity aligns with that purpose.
The motive is obvious and all over the comments. For you convenience, it can be summarized as gatekeeping. Start unpacking it and you will find ego (these are people at the top of a cloistered social pecking order who suddenly see that order threatened), entitlement (their writing reflects a belief that they own mathematics and have the right to dictate how it's done by others) and economics (academic funding follows prestige; being outdone by machines undermines the prestige, hence the funding).
> and also ignores the fact that your quote specifically targets the frontier labs, not LLMs or computational tools in general
That's like saying "your opposition to autonomous weapons ignores the fact that they specifically target [INSERT FAVORITE TARGET]". I ignore what's irrelevant.
If Ken Griffin had not chosen to donate $3 billion to CMU [1] and instead had spent that money on tokens to solve a bunch of open problems [2][3] using publicly available LLMs, do you seriously believe their reaction would have been more positive?
> The group (https://agmai.org/) comprises of world-renowned mathematicians, including several Fields medalists,
I am well aware of who they are, thank you very much.
> who have nothing left to prove.
In the old order now under threat.
> If they felt like it they could quit mathematics today and take private sector jobs paying far more than their professor salaries.
A common delusion among students and maybe even some professors who've never set foot outside academia. The authors of TFA know better, of course. Their current conditions amount to enjoying a comfortable living while pursuing their favorite hobby full time. In the private sector they would have to justify their salary by (gasp) working on something (OMG!) applied with a reasonable prospect of (would you believe it?) meaningfully contributing to their employer's (pardon my French) bottom line. A revolting thought, and prospective employers know that too. Both parties know it wouldn't work.
> the working group's various recommendations simply aim to ensure that computer-generated mathematical activity aligns with that purpose
Math is a tool. It's what people do with the tools after they are invented that matters - and with mathematical tools sometimes that wait is hundreds of years.
The mathematics community has found that the tools have value long after being invented if the inventor leaves his tools in a specific format. The community is doing its best to preserve that - not because they want less toolmakers, but rather because they want the tools to be useful when they become needed.
Yes. It’s remarkably flagrant, and it’s an example of an academic mindset that’s likely been holding progress back across multiple fields. It’s about to be broken rather badly, and it will show where progress has been stifled.
The mathematical community isn't really a single entity. It's weird to call that a monopoly.
An (obviously not 1-to-1) analogy is Amazon uses their platform sales data to push out almost identical AmazonBasic branded products. Small businesses can't compete, because Amazon has the economy of scale.
I am not saying that what Amazon doing is "wrong" per se, but I bet if small businesses have a voice they'd also ask Amazon to stop blatantly coping their most popular products.
It'd be weird to say that "the small businesses wants a monopoly on [insert product]."
Maintaining the existing standards is in fact a form of protection from AI disruption. They are asking the AI companies to follow their norms, instead of them having to conform to new norms created by AI. They don’t want to have to change the way they do things, understandably!, and are asking the companies to accommodate their way of life.
New norms are only worth adopting if they are clearly better, and that is far from obvious for whatever norms the proprietary AI companies are trying to push. Also their letter specifically targets proprietary AI companies, not computational tools in general which mathematicians do use when they advance mathematical understanding.
Better for who? If I don’t care about the welfare of mathematicians and just the advancement of mathematics, would AI doing the math not be better for me?
As well, if the open models were as good as the proprietary ones, do you think they wouldn’t still have complaints?
Is “gate keeping” a new toy for ai bros or something? Peer review has a purpose… You guys should try to understand systems before you celebrate their death.
I freely admit I am not an academic. I just note that there is a lot of negative sentiment about people trying to get stuff published, and a lot of it looks from an external perspective to be elitist gatekeeping.
Please read that again, I am not on the inside here, so anything I say is just based on reading stuff from aspiring academics and articles in that mode. To my amateur eyes, there seems to be a problem in the academic world, of some sort, but how big it is, or isn't, I don't know.
Is that ok?
So, assume I, a complete amateur, no PhD, no connections or academic record to speak of (I have an MSc only, from the 90s, in a third world country) thinks I've discovered something, how do I go about "publishing"?
To me, I could easily see value where I self publish somewhere, where other similar non-connected people can see, crowd source the verification (basically peer review) and if it turns out to be useful, it'll be used.
Maybe it would be easy for me to get it formally published, but I doubt it.
First of all if you think you had discovered something in a pre-AI world, there’s a 99% chance it’s some triviality or wrong. Mathematics is extremely rich and people spend their entire lives thinking about it. Undergrads can’t comprehend what the grad students are studying. Grad students can’t comprehend what the post docs are doing. And it just goes and goes.
In a post-AI world there’s a mountain of high quality work being published by people who actually know what’s going on. You can probably make a discovery with chatgpt right now, but why should someone who’s got a ton of things to do already read your article? You’re free to put it online, but why should you be free to force someone to read something?
Second of all, it’s extremely aggravating to read all these software people criticizing mathematicians, who almost universally have low salaries and who chose to go into this profession because they care about contributing to human knowledge.
>In a post-AI world there’s a mountain of high quality work being published by people who actually know what’s going on. You can probably make a discovery with chatgpt right now, but why should someone who’s got a ton of things to do already read your article? You’re free to put it online, but why should you be free to force someone to read something?
But aren't you literally just proving what I'm saying? The original argument was that everyone should have to post through peer reviewed journals, I said that is probably gatekeeping too far, and your response above just confirms my suspicions.
> Second of all, it’s extremely aggravating to read all these software people criticizing mathematicians, who almost universally have low salaries and who chose to go into this profession because they care about contributing to human knowledge.
I probably agree with you, except I have never and am not criticising mathematicians. I am only saying that there are quite a lot of reports around that the academic world is fierce and that sometimes there are high barriers to getting something recognised.
No idea if it is true or not, but my personal opinion is that it isn't going to matter much in the future. If it exists, then people will work around any gatekeeping.
> One of the foremost tasks of art has always been the creation of a demand
which could be fully satisfied only later. The history of every art form shows critical epochs in which a certain art form aspires to effects which could be fully obtained only with a changed technical standard, that is to say, in a new art form. The extravagances and crudities of art which thus appear, particularly in the so-called decadent epochs, actually arise from the nucleus of its richest historical energies. In recent years, such barbarisms were abundant in Dadaism. It is only now that its impulse becomes discernible: Dadaism attempted to create by pictorial—and literary—means the effects which the public today seeks in the film.
> Every fundamentally new, pioneering creation of demands will carry
beyond its goal.
Edit: to put a finer point on it, AI art like this is, through it's rigorous application in almost every previous medium, showing that there is a demand and energy for a new art form that depends on the changing technical standard and capabilities that AI has introduced. I don't know what it will be, but you can see some of the edges of it with how it is making waves in all the other mediums.
reply