The labs are spending huge money collecting "multisensory, perception-action coupled training data" (eg, there is the one in NY that gives you free cleaning in return for video data from the cleaner).
Edit: The Gemini Robotics blog post has a video of it tying knots too. That's pretty good.
You can point to the bitter lesson to support your claim, but on the other hand I can point to the massive investment in capital and time to get self-driving cars viable to support my more bearish view.
Yesterday evening I rode 12 miles in a Waymo actively dodging pedestrians and obstacles dynamically in an open ended environment. Multiples better experience than the two Ubers I had later that evening.
I'm not talking about the state of today's self-driving cars, I'm talking about how we got to here. Also don't forget the overarching claim regarding lack of sensor and actuator fidelity in the parent comment; self-driving cars in contrast have expensive LIDAR in them and we still don't think camera-based self-driving cars are safe enough.
I don't know how you can read my comment, not respond to my comment saying that today's self-driving cars use LIDAR, and continue to reiterate your point.
I don't think I was clear and explicit, it being tedious to write, and I apologize for that. I also apologize for shifting the goalposts as I had not written out my own position, which is not exactly in the "grandparent commenter"'s position (that I had not previously given enough attention understanding), but it is also not in agreement with yours. I don't mean to say that we are not presently in the "bitter lesson" (your idea of what the bitter lesson says) regime. I definitely think that a lot of progress can be done right now by emphasizing the humanoid robotics platform as a foundation. What I mean to say is that I don't know if that platform with the hardware we have today is sufficient for parity with human housekeeping tasks in the domains that we wish it to have parity. The bitter lesson itself (not your understanding of it) is in fact silent on this as it is in relation to feature engineering, where it is a clear point, but you seem to be adapting it uncritically wholesale to mean something more than what it is written about. My position is that, it is unclear whether today's sensor platform is sufficient for parity. It is less strong than the blog post author's "Why Today’s Humanoids Won’t Learn", it is a "We can't say whether or not today's humanoids will learn", but it is something that also contradicts a "the bitter lesson means today's humanoids will learn" thesis.
The self-driving car supports my claim, because after so much investment in capital and time, we ended up with a car with comparatively expensive LIDAR sensors as our preferred platform.
Tesla's entire fleet runs on raw cameras. Including the driverless Robotaxi vehicles - which are basically a 1:1 match to how Waymo operates.
Plenty of hecklers were saying "you can't self-drive on cameras", and some still try. But Tesla's self-driving on cameras, and it seems to work fine. While Waymo's self-driving on fat sensor stacks, and it also seems to work fine. Sensors don't seem to be a differentiator of self-driving performance.
I don't think anything about self-driving tech supports your claim. Tesla was bullish on AI all the way, and Waymo has also shifted towards highly integrated end to end AI. It's the AI advances that make self-driving tractable - not anything else.
It is amusing to observe that the tech marketing of today are milking the shit out of this trick.
The trick being to tread continuously through some non-obvious happy path. And average people will be convinced that you really have some breakthrough tech.
But hey, this is not something new. Magicians were taking advantage of such things for centuries ..
1. Make some thing that work in very limited of amount of real world cases
2. Deploy it somewhere where it won't encounter things it won't handle.
3. Market the shit out of the above fact and how well it work there.
4. Let the naive population who have a tendency to take one look, and imagine how it will automatically progress to some arbitrary influx point.
5. Get a lot of funding from people in point 4 and feed it to point 3, and keep going.
Why couldn’t this trick be pulled off two decades ago?
After all autonomous vehicles has been well funded research since the 1980s the DARPA grand challenge being one of the previously most important benchmarks.
I think you might just need a history lesson friend
Action LLMs work by generating text underneath. Just some higher level software interprets the text generated and do some action. So the immediate inference result is still text.
Obviously there is a lot of variety in architecture, but in the prototypical example there are vision and languages encoders and an action decoder which decodes direction into action steps. Eg, Hugging Face SmolVLA:
> Specifically, the VLM processes sensorimotor states, including images from multiple RGB cameras, and a language instruction describing the task. In turn, the VLM outputs features directly fed to the action expert, which outputs the final 3 continuous actions.[1]
Or NVidia's GR00T N1:
> A diffusion transformer (DiT) processes the robot’s proprioceptive state and action, which are then cross-attended with image and text tokens from the Eagle-2 VLM backbone to output the denoised motor actions.[2]
Depends entirely on VLA arch. Some have dedicated action diffusion heads that work in a standalone non-text action output space. Much like an LLM can either use an external TTS or have audio output heads attached to it directly for native S2S.
But your entire premise is wrong regardless of that.
Even if VLAs were forever bound to outputting text, you'd have to prove that they're fundamentally incapable of emitting text that maps to useful action sequences. No proof of that whatsoever - and plenty of empirical evidence suggests otherwise. Even non-specialist LLMs like ChatGPT are getting better at controlling robots and navigating 3D environments, if slowly.
>But your entire premise is wrong regardless of that.
You don't understand what I am saying. The crux of your misunderstanding is here
>emitting text that maps to useful action sequences
If you have a static mapping from text to action, then you are throwing away all the advantage of using an AI. The whole point of AI is that you can get an output from an input without explicit mapping. So If you use explicit mapping anywhere in the chain, then you lose most of the advantage of using the AI.
So if your hardware, physical vocabulary is limited, like move left/right/up/down then what you say could work. But something that have the dexterity of a human form, this vocabulary is nearly infinite. You won't be able to use explicit mapping there.
You can literally have an LLM output target joint angles. As text. To be decoded by an explicit decoder, and executed by the robot. Some early VLAs did exactly that.
Your entire premise is wrong.
Modern action decoders are different, and usually take the form of neural networks trained end to end jointly with the rest of the model. Not fundamentally more expressive, just more in line with what we want.
I'm repeating "what you claim to be impossible was done 3 years ago and was already replaced with better versions of the same idea and you are hilariously out of touch".
As I suspected you are fooled by this video and imagine it to be capable of much more than what is shown. This video is pretty non-marketing and is quite straight to the point. But that does not prevent you from being awed!
So What is LLM is used here for? It is used for mere translation between different robots. So it is mostly symbolic translation.
What I am talking about is to translation LLM inference directly to movements. For example, if you ask an LLM, how do I open the microwave door? It will list the steps. I am talking about a system that can go from "put the thing in the microwave", to action steps, without having to never once demonstrate it physically, and do it just from LLM inference.
In short, the way LLMs used here is not (categorically) the way I was asking about.
That is a straight forward task. I am talking about using an LLM to come up with a sequence of complex steps that does not have an intermediate textural representation.
So it should be something like, "put back this slipped cycle chain back on sprocket"..
The thing literally has a diffusion "action expert" sit in the same attention system as a pre-trained VLM. And the VLM itself is ALSO trained to generate raw actions as a part of the training recipe (the first paper) - it just doesn't do it at inference time. What the "action expert" does is parallelize the action generation process - based on VLM's internal states.
It's exactly the thing you claimed to be impossible. Described in detail in a paper from 2025. What's your excuse?
I have not overlooked anything. I had imagined that this "mapping", to have any power, would also need to be handled by an LLM (action expert). But here is the problem with that. That would not be as "intelligent" as an LLM....And you can't make it as smart as the LLM because there is not a similarly huge training data on which LLMs are trained on..
The backbone of the VLA there is literally a pre-trained Gemma model. And a small one at that.
You already downgraded your claims from "LLMs are irrelevant to robotics" to a measly "you can't train a useful robotics LLM because there's not enough data". And you say that while looking at an LLM that was pre-trained on all of internet scraped and only then reused for robotics.
Both the pool of robotics-relevant data and the performance of foundation model LLMs grow over time. All the companies that are serious about robotics are serious about scaling up data collection.
I'm not going to claim that this "LLM core" approach is the best approach to AI robotics possible - but if you're betting on it failing outright, you're going to be fighting uphill.
Modern robotics is, at its core, not a hardware problem. It's an AI problem. We have plenty of headroom in the hardware - what we don't have is an AI good enough to utilize it. We don't know the practical limits of current hardware because we can't make a robot AI that would make the hardware a meaningful bottleneck.
Today's robots don't fail at tasks because they have poor fingers. They fail because they don't know how to perform those tasks. If you put an effort into solving that? You get demos like: Gemini Robotics 2 tying a garbage bag. Take one long look at that and think of manual dexterity.
Human body is crude and suboptimal in a thousands different ways, and all of it is salvaged by advanced intelligence.
Yeah, look carefully at that demo of tying the garbage bag strings: it's done in a very peculiar style that suggests a very specific, very precise, "algorithm" taught in an imitation learning session, which has no chance to transfer to other tasks, or even other garbage bag strings.
Did the past decades of AI research teach you absolutely nothing?
Every time you see something that "suggests a very specific, very precise, "algorithm" taught in an imitation learning session"? Scale the imitation learning up x10, x100, x1000, and it suddenly generalizes!
I'll be honest: I don't see what you see. I don't see anything that would suggest this algorithm is so brittle there's zero transfer to "even other garbage bag strings". AI robotics isn't innately brittle like conventional robotics is. But even if you are, somehow, completely right on that? Teach a hundred "very specific algorithms" like this - and watch them fuse into a manifold of algorithms that can be applied to different problems as needed.
And that is what you need. If an algorithm for "tie a garbage bag with current generation robot hands" exists and can be learned by an AI, then the gains from getting better AI are far from exhausted. The limits of robotics are the limits of AI.
This is why every AI robotics company is saying "we need more data". They understand what they're dealing with. They looked at the scaling laws and went "robotics isn't magic, that curve applies to us too". I don't get what makes people see robotics as a special magic thing, that makes them look at the advances in robot AI and say "this is intractable" and not "this is hard". It's hard. We're getting through it though.
>> Did the past decades of AI research teach you absolutely nothing?
Before I put in the effort to reply in good faith I have to know: do you think we're going to have a conversation or are you going to fulminate and scold me like some kind of all-important authority (which I have to say you clearly are not)?
To clarify, I'm happy to have a curious and respectful exchange.
Blooms 2 sigma, like many studies that are thrown about, is not as much of a slam dunk as people claim that it is. Have you read the paper or any efforts to replicate?
It's been many months, and I thoroughly prefer my harness inside of nvim as my day to day development environment. Using Claude code or cursor makes me feel very removed from the code.
Exploring code, gathering context and tweaking prompts/giving guidance to the agent are very much enhanced by neovim.
The biggest boon has been the fact that agents make customizing neovim a lot easier. Writing new bindings, config, and even building novel plugins.
Here's a few that I built that fit into my workflow:
- a file picker that renders in your current window (like oil) and uses more intelligent signals for ranking (like frecency) https://github.com/dlants/needle
Recently added support for scripts (like Claude code workflows) and been iterating on the UI for that a bunch.
I also ended up wanting other customized tooling - a more streamlined way to grep, find files and review code that my agent has written. So I wrote a few plugins for that : needle (finder with UI and sorting functions that suit me better), shuck (interactive grepper that has a workflow around refining grep commands) and glean (a review tool that lets you mark parts of the code as seen, leave comments, view diffs commit by commit or collapsed, etc). https://github.com/dlants/dotfiles/tree/main/nvim/lua
These are all in various states of experimental and mostly just for me, but a few of my coworkers and friends have been using magenta and like it.
> Stocks would become eligible for the index after six months rather than 12 months. The requirement to have a minimum Investable Weight Factor of 0.10 (roughly at least 10% of shares publicly floated) would be dropped. Companies would not be required to demonstrate profitability.
> Still, S&P Dow Jones reminds market participants that the proposed changes would apply only to index eligibility. The actual inclusion of new constituents remains entirely at the discretion of the index committee.
I guess the hope is that combining two sub-par coding models (xAI's grok + cursor's composer) and combining the data they have access to, they can build something that can compete with OpenAI / Anthropic in the coding space...
I guess I kinda see it... it makes sense from both points of view (xAI needs data + places to run their models, cursor needs to not be reliant on Anthropic/OpenAI).
I think I don't see it working out... I just don't see an Elon company sustaining a culture that leads to a high-quality AI lab, even with the data + compute.
Have to call out that comment about grok code being sub par. I used it exclusively when it was free in Cursor and have nothing bad to say about it. And that was months ago. I imagine it’s a lot better now.
Composer-2 is based on Kimi K2.5, but with extensive RL. Cursor estimated 3x more compute on their RL than the original K2.5 training run (some details in https://cursor.com/blog/composer-2-technical-report).
I used to hate on Composer 2 but I'm coming around to it. Opus for the big stuff and multi-file operations, Composer for all the small day-to-day IDE tasks works pretty good for me.
I'm going to be brutally honest but I have not found Kimi to be useful at all. It simply cannot compete with what closed models from Codex and Claude offers. I don't want to risk using a model outside the ecosystem and introduce variables as most of my workflow is baked into two to three large company models.
That's interesting, Kimi K2.5 used through KimiCode was comparable to Sonnet in my tests, and is an excellent alternative to Anthropic models
That being said, I noticed that Kimi being served through Openrouter providers was trash. Whatever they do on the backend to optimize for throughput really compromised the intelligence of the model. You have to work with Kimi directly if you want the best results, and that's also probably why they released a test suite to verify the intelligence of their new models.
Hey, thanks for responding. You're a very evocative writer!
I do want to push back on some things:
> We treat "cognitive primitives" like object constancy and causality as if they are mystical, hardwired biological modules, but they are essentially just
I don't feel like I treated them as mystical - I cite several studies that define what they are and correlate them to certain structures in the brain that have developed millennia ago. I agree that ultimately they are "just" fitting to patterns in data, but the patterns they fit are really useful, and were fundamental to human intelligence.
My point is that these cognitive primitives are very much useful for reasoning, and especially the sort of reasoning that would allow us to call an intelligence general in any meaningful way.
> This "all-at-once" calculation of relationships is fundamentally more powerful than the biological need to loop signals until they stabilize into a "thought."
The argument I cite is from complexity theory. It's proof that feed-forward networks are mathematically incapable of representing certain kinds of algorithms.
> Furthermore, the obsession with "fragility"—where a model solves quantum mechanics but fails a child’s riddle—is a red herring.
AGI can solve quantum mechanics problems, but verifying that those solutions are correct still (currently) falls to humans. For the time being, we are the only ones who possess the robustness of reasoning we can rely on, and it is exactly because of this that fragility matters!
> The argument I cite is from complexity theory. It's proof that feed-forward networks are mathematically incapable of representing certain kinds of algorithms.
Claiming FFNs are mathematically incapable of certain algorithms misses the fact that an LLM in production isn't a static circuit, but a dynamic system. Once you factor in autoregression and a scratchpad (CoT), the context window effectively functions as a Turing tape, which sidesteps the TC0 complexity limits of a single forward pass.
> AGI can solve quantum mechanics problems, but verifying that those solutions are correct still (currently) falls to humans. For the time being, we are the only ones who possess the robustness of reasoning we can rely on, and it is exactly because of this that fragility matters!
We haven't "sensed" or directly verified things like quantum mechanics or deep space for over a century; we rely entirely on a chain of cognitive tools and instruments to bridge that gap. LLMs are just the next layer of epistemic mediation. If a solution is logically consistent and converges with experimental data, the "robustness" comes from the system's internal logic.
Thanks for reading, and I really appreciate your comments!
> who feed their produced tokens back as inputs, and whose tuning effectively rewards it for doing this skillfully
Ah, this is a great point, and not something that I considered. I agree that the token feedback does change the complexity, and it seems that there's even a paper by the same authors about this very thing! https://arxiv.org/abs/2310.07923
I'll have to think on how that changes things. I think it does take the wind out of the architecture argument as it's currently stated, or at least makes it a lot more challenging. I'll consider myself a victim of media hype on this, as I was pretty sold on this line of argument after reading this article https://www.wired.com/story/ai-agents-math-doesnt-add-up/ and the paper https://arxiv.org/pdf/2507.07505 ... who brush this off with:
>Can the additional think tokens provide the necessary complexity to correctly
solve a problem of higher complexity? We don't believe so, for two fundamental reasons: one that
the base operation in these reasoning LLMs still carries the complexity discussed above, and the
computation needed to correctly carry out that very step can be one of a higher complexity (ref our
examples above), and secondly, the token budget for reasoning steps is far smaller than what
would be necessary to carry out many complex tasks.
In hindsight, this doesn't really address the challenge.
My immediate next thought is - even solutions up to P can be represented within the model / CoT, do we actually feel like we are moving towards generalized solutions, or that the solution space is navigable through reinforcement learning? I'm genuinely not sure about where I stand on this.
> I don't have an opinion on this, but I'd like to hear more about this take.
I've been developing an ai coding harness https://github.com/dlants/magenta.nvim for over a year now, and I use it (and cursor and claude code) daily at work.
Fun observation - almost every coding harness (claude code, cursor, codex) uses a find/replace tool as the primary way of interacting with code. This requires the agent to fully type out the code it's trying to edit, including several lines of context around the edit. This is really inefficient, token wise! Why does it work this way? Because the LLMs are really bad at counting lines, or using other ways of describing a unique location in the file.
I've experimented with providing a more robust dsl for text manipulation https://github.com/dlants/magenta.nvim/blob/main/node/tools/... , and I do think it's an improvement over just straight search/replace, but the agents do tend to struggle a lot - editing the wrong line, messing up the selection state, etc... which is probably why the major players haven't adopted something like this yet.
So I feel pretty confident in my assessment of where these models are at!
And also, I fully believe it's big. It's a huge deal! My work is unrecognizable from what it was even 2 years ago. But that's an impact / productivity argument, not an argument about intelligence. Modern programming languages, IDEs, spreadsheets, etc... also made a fundamental shift in what being a software engineer was like, but they were not generally intelligent.
> Fun observation - almost every coding harness (claude code, cursor, codex) uses a find/replace tool as the primary way of interacting with code. This requires the agent to fully type out the code it's trying to edit, including several lines of context around the edit. This is really inefficient, token wise! Why does it work this way? Because the LLMs are really bad at counting lines, or using other ways of describing a unique location in the file.
Incidentally, I saw an interesting article about exactly this subject a little ways back, using line numbers + hashes instead of typing out the full search/replace, writing patches, or doing a DSL, and it seemed to have really good success:
It's general-purpose enough to do web development. How far can you get from writing programs and seeing if you get the answers you intended? If English words are "grounded" by programming, system administration, and browsing websites, is that good enough?
$2700/mo is about 1/3 of an engineers' salary (cost to the business of a mid-level engineer in the UK)...
But, there's the time to set all of this up (which admittedly is a one-time investment and would amortize).
And there's the risk of having made a mistake in your backups or recovery system (Will you exercise it? Will you continue to regularly exercise it?).
And they're a 3-person team... is it really worth your limited time/capacity to do this, rather than do something that's likely to attract $3k/mo of new business?
If the folks who wrote the blog see this, please share how much time (how many devs, how many weeks) this took to set up, and how the ongoing maintenance burden shapes up.
- the relatively crude tactile and proprioceptive sensing apparatuses of robots when compared to humans
- the limited availability of multisensory, perception-action coupled training data
Genuinely curious!