I don't know enough about the specific models they're comparing against to say this definitively, but it looks to me like they're comparing their pre-trained models with others' post-trained models.
The metric upon which their 10x claim is based (bits-per-byte) is exactly the metric which is optimized during pre-training. Post-trained models are fine-tuned to optimize other metrics, which is known to be detrimental to performance on bits-per-byte evaluations. So bits-per-byte evaluations will always make a pre-trained model look favorable in comparison to a comparable model which has also undergone post-training.
Can someone confirm whether the models they are comparing against (DeepSeek V4, Kimi K2, and Nemotron 3 Ultra) have been post-trained?
> “Base models” are pretrained models that have not yet undergone reinforcement learning, SFT, or other post-training. They are highly sensitive to prompting, making sampling-based evals unreliable. Instead, we measured bits-per-byte loss on heldout data, which does not suffer from prompt sensitivity and smooths measurement of otherwise emergent abilities. As a side note, we were surprised that Nemotron 3 outperforms DeepSeek V4 Pro across the board but found this to be consistent across domains and inference engines. This might indicate that Nemotron’s weak performance on benchmarks after RL is due to weaker post-training, but the pretrain was ahead of Chinese open-weight competitors.
One of the reasons why I have been thinking and writing this topic is because there is indeed a malaise among the team, which is along the lines of "AI agents are blowing up coding everywhere, what place does a compiler have in this brave new world?" If compilers truly are pointless, then I would like to update on this and save people time and grief!
But I also don't think compilers are pointless. In this post, which looks at the maximalist AI situation, the compiler resurrects itself as the verifier: the two types of systems do very similar things! You can also defend compilers on more banal grounds: not everyone has a ton of kernel engineers to hand optimize your kernels; token efficient ways of getting good baselines is useful; there's more to ML than transformer models and compiler leverage is really useful there. There's also a historical point which is that torch.compile... by all objective metrics, has really been quite successful!
I do agree it's hard to tell the future. These days, I ask myself, "Do I think this is likely to happen in six months." I don't think compilers die in six months. Ask me again in six months :)
The folks submitting papers to SIGPLAN and PLDI seem to be the opinion use of AI technology in compilers will only increase merging both approaches, of natural language as frontend, and backends.
I think many that don't follow compilers haven't yet grasp how much there is already there, from machine learning based optimisation passes, JIT compiler heuristics, and so on, which never have been deterministic to start with.
I can see how LLMs may make compiler development seem futile when it comes to optimizing the local graph structure (ie kernel-sized chunks of the computation).
But optimizing the global structure -- things like defining the boundaries between kernels, the lifetimes of allocations (or when/how to reuse buffers), and scheduling/parallelizing operations -- seems inextricably tied to the type of traditional program analysis used by compilers.
Even tiling/layout decisions, which you mentioned can be optimized by hand or by an LLM in the context of a single kernel, must be optimized in the broader context of the computation to minimize copies/relayouts between kernels.
I think part of the problem is that massive models make the compiler problem much less interesting, since global program optimization reduces to local kernel-level optimization.
The point is there are limits on how much better the compilers can get, without modifying the language to be closer to the target environments. And there are multiple target environments, so a one-size-fits-all solution is not really possible. You have to compromise somewhere - you can’t have a single high level language that automatically produces maximally efficient code for every target, because the targets are not general-purpose computing environments, they have significant constraints.
If what you’re looking for is a way to avoid having to get into the operational details of the targets, and just provide a high level description that somehow gets figured out, then an LLM-based solution is almost certainly going to be your best bet.
I skimmed the doc and my impression is that your second listed interpretation -- DeepSeek is pausing investment because of a leak -- is the more correct one.
There's quite a bit of confidential information in the doc about the company and how it's positioning itself going forward to compete with US labs. I'd imagine they're not happy at all with this being leaked and are withholding investment as a punitive measure.
Not to mention the other interpretation seems illogical -- why would you pause fundraising if your perception was that you lacked resources compared to your competitors?
Because money isn’t free and if you know there is a chokehold in supply, why raise money ar current valuation when things are getting better by the day?
I've found breathing exercises to be effective for the duration of the exercise, but I'm more interested in the possibility of training myself to adjust my respiration patterns over sustained durations. Would it be beneficial -- or even possible at all -- to adjust my body's default/subconscious breathing patterns to match those mentioned in the article?
Tangentially related, are there any wearable devices that allow for high resolution respiration monitoring? I'm imagining some measurement of lung expansion over time (probably at least 10 Hz) so that I can quantify the deepness/shallowness of my breaths as well as the phase of inhalation/exhalation cycles.
Anectata, but I found that active wearables can help with initial motivation, but long term they become chore or even stressful when forced. I use them only for passive long trends tracking.
Our brains trick us to breath on defaults adjusted to our surroundings.
I have read that skilled mindfulness practitioners maintain constant awareness of their breathing pattern throughout all other waking activities. Something to aspire to perhaps.
I think it is helpful to practice autonomic control throughout the day. If you are riding the bus or sitting in a circle or doing this or that and you have the spare bandwidth to think about it. It becomes a habit.
People don’t like anxious people so it is part of charisma development like you do training for spiritual leadership or relational healing. Less anxiety means more tolerance for risk, ambiguity, etc. it is not “put it all on 7“
There's also a level above that, where you're aware of what is aware. I find this mental state to be even more calming/grounding than being aware of breathing but I'm not always able to shift myself into it. Being aware of breathing feels much easier/natural to me whenever I'm able to remind myself of it, which already provides quite a noticeable effect on how I'm feeling and reacting to whatever is happening in that moment.
Additionally, there's a practice called "walking meditation" [0] that can also be useful to practice this area of skills.
Finally I understand why meditation hippies kinda seem aloof when you talk to them. Turns out they are trying to multi-task some kind of awarenessmaxxing while speaking to you. The more you know!
I always thought that was part of their weirdness and maybe even some personality trait that led them to this sort of thing, but knowing it's an active choice makes it even weirder somehow.
Sometimes it is great to stomp on people’s lines but slowing down your responses half a beat is beneficial in most situations. There is often a first mover disadvantage.
Awareness of breathing does not mean controlling your breathing, it just means noticing the sensations associated with it. Breathing can be incredibly pleasant!
> Would it be beneficial -- or even possible at all -- to adjust my body's default/subconscious breathing patterns to match those mentioned in the article?
Common physical reflexes, autonomous responses, and subconscious regulation, are there as aids to us. The fact that they are not universally beneficial is one of the purposes of having higher level control. Not to universally suppress responses, but to notice and cope when they misfire.
It would be interesting to have a map of breathing patterns across a wide variety of situations, to identify the range of situations where prolonged exhalation is adaptive.
My guess, based on the common reflexes of mouth clamping and breath holding before great physical exertion, is that prolonged exhalation is part of an adaptive psychological orchestrator for when we prepare to take on something difficult, risky (but necessary), or that needs a fast strong response.
Our fast acting emotions, and slower acting moods, are similar guides. Patterns of stimulus and response from our baseline physiology and psychological, that we absorb into our higher level operation, as generalized guides for analogous responses to contexts at higher abstraction levels.
With minor maladaptive responses inevitable, if we don't pay attention. And severe maladaptive responses often ingrained as overcompensation for situational or developmental traumas.
The craziest thing I noticed about a breathing pattern and risk taking was when a murderer was in an interrogation room with a police officer when after they couldn't find his gun; he had stowed it on his ankle. The suspect took a deep inhale after reaching for his gun while the officer was focused on the computer screen in front of him, exhaled and swiftly aimed at the officers temple and fired. Then he broke out of custody and was caught shortly after.
Kind of a backwards take, both in your interpretation of this announcement and the company in general.
The point of this announcement is to draw attention to the fact that the currently hyped topic is what they have been working on since their inception. If anything, it gives off a Schmidhuber-esque 'actually it was me who invented that' vibe. But trying to retroactively claim credit for the hype is nowhere near the same as following the hype.
As for your impression that the company is more generally hype-chasing, I'm really not sure how you would come to that conclusion. At the time of their founding, chatbots were the hype on the product side and model scaling was the hype on the research side -- topics they have largely eschewed. They instead were founded with a focus on evolutionary and collective intelligence and have maintained a fairly cohesive research direction ever since.
"For the past 300,000 years, Earth has had only one form of advanced intelligence on it: humans. With the recent advent of AI foundation models, some believe we are at the dawn of a new kind of intelligence. As AI continues to evolve, we may witness the proliferation of diverse intelligent lifeforms coexisting with us."
I am not even going to link more than one thing I think I've made my point
In multiple threads now you've claimed I was wrong and then ended the conversation without any assertion, claim, or argument I could rebut. You've ended those conversations with snarky comments clearly designed to shut down debate. This is not how hackernews benefits.
Edit: Nevermind, realized he's just an uneducated troll just looking for his kicks. Comment flagged.
I see what you're getting at, but determinism isn't the right word either. LLMs are fundamentally deterministic -- they are pure functions which output text as a function of the input text and the network parameters[1]. Depending on your views on free will, it could be effectively argued that humans are deterministic as well.
The concept you're touching on is the idea that LLMs (and humans) are functions which are inscrutable. Their behavior cannot be distilled into a series of logical steps that you can fit in your head, there are no invariants which neatly decompose their complexity into a few interpretable states, and the input and output spaces are unstructured, ambiguous, underspecified, and essentially infinite. This makes them just about impossible to reason about or compose using the same strategies and analysis we apply to traditional programs.
[1] Optionally, they can take in a source of entropy to add nondeterminism, but this is not essential. If LLM providers all fixed their prng seeds to a static value, hardly anyone would notice. I can't imagine there are many workflows which feed an LLM the exact same prompt multiple times and rely on the output having some statistical distribution. In fact, even if you wanted this you may just end up getting a cached response.
Let's be real, if you and I both ask claude to generate a feature on the same project, what are the chances that it spits out 100% replicated code? But if we are to build the project using a Dockerfile, we will get the same binary and the same image. Products around LLMs are non deterministic unlike compilers.
I can assure you that a fully deterministic and equally effective claude is possible to build. And yes, that would mean identical prompts would yield 100% identical output 100% of the time. It would still make the occasional logical or factual error, but it would do so deterministically. Would this solve any of the problems with building reliable programs using LLMs?
it's nondeterministic because we chosen it by having higher 'temperature' in settings. I bet if you run open weights model with temperature 0 and on the same device the same prompt and turn off parallelism you will have more deterministic result (excluding some floating point operations).
> Optionally, they can take in a source of entropy to add nondeterminism, but this is not essential. If LLM providers all fixed their prng seeds to a static value, hardly anyone would notice
Everyone added /dev/random to their offerings, so every LLM tools for coding are non deterministic.
The guy writing a thumbnail pipeline isn't getting petabytes (exabytes?) of storage to cache all videos from the past week in their entirety. If this quantity of data is being stored, it's being stored deliberately and at significant cost.
The other explanations here don't explain the long delay between the start of the investigation and the release of the footage.
Yes, storing customer data is what we'd expect from Google and yes, the FBI can coerce Google to provide this data for their investigations.
But it does not take a week for Google to find a file on their servers.
My hunch is that Google initially tried to play dumb to avoid compliance, as to not reveal they do in fact retain customer data.
They had a plausible excuse as well -- the owner had no subscription so they don't store the data -- and took a gamble that this explanation would suffice until the situation resolved itself.
I suspect that authorities initially took Google's excuse at face value, since they parroted this explanation to the public as well.
As pressure mounted on authorities to make some headway on the case, they likely formally exercised whatever legal mechanisms they have at their disposal to force Google's hand, and only then was the footage released.
This is a wild claim. I would think criminal charges for something like obstruction would be possible if Google intentionally hid this from investigators for up to a week. That could result in the difference between the victim being found alive or not.
The implication that OpenAI is a YC company in the same sense as the other listed companies is somewhere between misleading and dishonest. Even more distasteful to show founding teams for all the others, then just Sam for OpenAI.
Of all Schmidhuber's credit-attribution grievances, this is the one I am most sympathetic to. I think if he spent less time remarking on how other people didn't actually invent things (e.g. Hinton and backprop, LeCun and CNNs, etc.) or making tenuous arguments about how modern techniques are really just instances of some idea he briefly explored decades ago (GANs, attention), and instead just focused on how this single line of research (namely, gradient flow and training dynamics in deep neural networks) laid the foundation for modern deep learning, he'd have a much better reputation and probably a Turing award. That said, I do respect the extent to which he continues his credit-attribution crusade even to his own reputational detriment.
I think one of the best things to learn from Schmidhuber is that progress involves a lot of players and over a lot of time. Attribution is actually a difficult game and usually we are only assigning credit to those at the end of some milestone. It's like giving a gold medal to the runner in the last leg of a relay race or focusing only on the lead singer of a band. It's never one person that does it alone. Shoulders of giants, but those giants are just a couple of dudes in a really big trenchcoat.
Another important lesson is that often good ideas get passed over because of hype or politics. We often like to pretend that science is all about the merit and what is correct. Unfortunately this isn't true. It is that way in the long run, but in the short run there's a lot of politics and humans still get in their own way. This is a solvable problem, but we need to acknowledge it and create systematic changes. Unfortunately a lot of that is coupled to the aforementioned one.
> I do respect the extent to which he continues his credit-attribution crusade even to his own reputational detriment.
As should we all. Clearly he was upset that others got credit for his contributions. But what I do appreciate is that he has recognized that it is a problem bigger than him, and is trying to combat the problem at large and not just his own little battlefield. That's respectable.
It's a bit of an aside but I believe this is one reason Zuckerberg's vision for establishing the superintelligence lab is misguided. Including VCs, too many people get distracted by rock stars in this gold rush.
Just last week I said something inline with that[0]. Many people conflated my claim that Meta has a lot of good people with "Meta /is/ winning the AI race". I just claimed they had some of who I think are some of the best researchers in the field, but do not give them nearly the same resources or capacity to further their research that they give to these "rock stars". Tbh, the same is true for any top lab, I just think this happens more at Meta because Meta is so metric and rock star focused.
So I agree. The vision is misguided. I think they'd have done better had they taken that same money and just thrown it at the people they already have but who are working in different research areas. Everyone is trying to win my doing the same things. That's not a smart strategy. You got all that money, you gotta take risks. It's all the money dumped into research that got us to this point in the first place.
It's good to shift funds around and focus on what is working now, but you also have to have a pipeline of people working on what will work tomorrow, next year, 5 years, and 10 years. The people are there that can do that work. The people are there that want to do the work. The only thing is there's little to no people that want to fund that work. Unfortunately it takes time to bake a cake.
Quite frankly, these companies also have more than enough money to do both. They have enough money to throw cash hand over fist at every wild and crazy idea. But they get caught in the hype, which is no different than an over focus on the attribution rather than the process or pipeline that got us the science in the first place.
Also, it reminds us that the powerful write history. But history can be rewritten as the balance of power shifts. I imagine the world will hear all about China's contributions to the field if they continue their ascent.
The metric upon which their 10x claim is based (bits-per-byte) is exactly the metric which is optimized during pre-training. Post-trained models are fine-tuned to optimize other metrics, which is known to be detrimental to performance on bits-per-byte evaluations. So bits-per-byte evaluations will always make a pre-trained model look favorable in comparison to a comparable model which has also undergone post-training.
Can someone confirm whether the models they are comparing against (DeepSeek V4, Kimi K2, and Nemotron 3 Ultra) have been post-trained?