Does this actually work for anybody? In my experience, when you use cheaper models as subagents, the more expensive parent agent ends up reading all of the subagent's work and surrounding code into context anyway to review it. So you end up paying more, not less.
Plus the subagent's work is often subpar and the parent ends up requesting multiple revisions. Each back and forth burn unnecessary tokens as well as my time.
Where it might work well is for separate task categories like image generation - get the parent to ask for an image from another model that is better at this aspect
When I am making agents do a hard thing, I use Opus as both parent and child. I make the child review its code extensively, and then still make the parent read the work and review once again. It is much faster and cheaper than just rolling with no extra reviews, and then in a day I have fifty subtle bugs.
> If humans remain in control of it, then how can it possibly lead to anything other than extreme oligarchy?
Depends on the distribution of such AI. For example, if everyone on Earth has their own, personal superhuman AI, it would not lead to extreme oligarchy. If only a handful of humans control superhuman AI, it would.
The linked tweet about remote vs in-person is hilarious to me.
I've never witnessed anything else affect intensity in my teams as negatively as forced RTO did.
I mean of course it is fully dependent on personalities and roles, and totally anecdotal, but in my experience, work was certainly *much* more intense when people had conditions for deep focus. I was literally just thinking about how slow everything feels nowadays, with everybody struggling to get focus time in a crowded office full of distractions.
I hear this meme of "Opus writes better code than me" repeated a lot. I am absolutely baffled by it.
I use all the latest and greatest Anthropic (and OpenAI) models, and interact with them probably on average 10h a day. I experiment with changing effort levels, with different ways of prompting and guardrailing, etc. Giving more instructions, giving less instructions. Doing heavy planning, just giving vague ideas.
I've done everything from fully vibe coding massive projects, to using LLMs regularly in my day job and reviewing every single line they produce.
Unless you are *really* on top of your agents, they inevitably end up writing an absolute mess. Not only does the code become impossible for humans to reason about very quickly, it actually becomes difficult for agents to reason about it. It turns into a total garbage in = garbage out situation.
Have you honestly reviewed a vibe coded project after it has been going for a while, even one written by current SOTA models, and thought "yep, this is great code"?
> Unless you are really on top of your agents, they inevitably end up writing an absolute mess.
It's like the magic broomsticks from Fantasia's Sorceror's Apprentice sequence. Great at hauling buckets but if you don't keep a close eye on them they will fill your workshop with water.
Like what matters is:
- Does it do the thing it's supposed to do
- Does it do it with a reasonable amount of resources
- Does it do it without any obvious security flaws
Yeah, if they were nuclear codes you'd want to be really really sure and you'd probably want to check every line yourself. But if you're making a habit tracker or 95% of apps out there, it really doesn't matter.
There is definitely some skill in piloting the LLM, where you have an advantage if you understand what "good" is - but I'd argue it really isn't that necessary in most cases. McDonald's has had a lot of success selling food that most chefs would say isn't that great, but it fills that need at that moment.
I don't think this is inevitable. How do you define an absolute mess? Whatever you come up with, give the agent instructions to not do that. Then audit the code to make sure it's not happening.
> Not only does the code become impossible for humans to reason about very quickly,
What makes it actually impossible? It's using logic that is beyond human comprehension? It's using the programming language in a clever way no human would expect? Have you tried instructing it to code in a way that's more comprehensible to humans, e.g. literate programs?
> it actually becomes difficult for agents to reason about it.
> Whatever you come up with, give the agent instructions to not do that. Then audit the code to make sure it's not happening.
Yes, this is what I mean by saying you need to stay on top of the agents
> What makes it actually impossible?
A combination of wrong abstractions, lack of abstractions, assuming some key requirements were just implementation details and thus are safe to refactor, assuming some implementation details were key requirements and thus must be protected at all costs, etc etc.
In practical terms, it's just volume of code that is spread extremely wide, duplicated with subtle differences, and just organized extremely confusingly.
> This I have not seen. Do you have an example?
If you try to vibe code longer term projects, you very often start getting completely nonsensical explanations from agents about why they are doing things. They reference requirements which are not real (based on what they interpret from the messy code), they write thousands of test cases that are redundant, or test things which are implementation details.
Eventually even trivial changes become extremely slow due to lack of structure, due to fresh agents being confused by contradictions introduced by previous sessions, and due to test suites that take hours to run.
Yes. A project I am working on now the code quality it is producing I would put at less than ten people I have ever met could do, and they would struggle with the OCD tedium induced.
There is clearly a trick to coaxing these results, and a major factor is to treat the model like it is something you fully expect to get it right because it knows how to do it, not you micromanaging it into compliance.
> Not only does the code become impossible for humans to reason about very quickly, it actually becomes difficult for agents to reason about it.
Story of many enterprise applications built only by humans. There is a good reason that frequently devs either refactor their codebase or do a complete rewrite.
True. In human built applications, the problems really start manifesting when the original code owners/maintainers leave, effectively turning their work into legacy code.
But vibe coded projects are effectively instant legacy code, due to lack of real ownership. So it's kind of a bigger problem.
> We are in fact seeing software improvement, in the form of fewer bugs and better performance.
Anecdotally, completely disagree. Maybe this is nostalgia, but I perceive a distinct and massive drop in software quality over the past few years.
Worst offenders for me are the poster children for LLM-based development. I run into bugs and performance issues with both Claude and Codex literally every day. I don't think I'm an outlier here, either.
- more bugs becoming visible that were hidden within the software we use daily
- more bugs (in absolute terms) as more people create more software in a very short amount of time (regardless of their professional training)
- less bugs in existing software as the developers and dev orgs of existing projects is capable of automating bug hunting, test case creation, ... in essence: They get a multiplying effect out of AI regarding their capabilities.
In short: Actual developers with actual experience (the more, the merrier) can do better work while the barrier to enter "the market" for untrained individuals is a lot lower resulting in a huge amount of new software whose "success" is to be determined.
> - less bugs in existing software as the developers and dev orgs of existing projects is capable of automating bug hunting, test case creation, ... in essence: They get a multiplying effect out of AI regarding their capabilities.
I'm not convinced there are more than a handful of orgs in the world who are disciplined enough to achieve this. Or organisationally willing to increase their QA Eng headcount sufficiently to achieve it.
There are so many stupid little bugs in software these days that should be straightforward to fix, but just aren't fixed. Visual quirks and rendering issues. Notification bells that don't clear properly. Formatting problems. And vibecoded apps like Claude Code themselves are the biggest perpetrators.
To add some related evidence to that, I browsed this report[1] for other reasons the other day, but it stood out to me that _100%_ of Forbes Global 2000 tech leaders said "their organization has experienced some form of AI-related downtime".
Dev being reckless with LLMs are going to ship faster than the one using it responsibly to improve quality, so it would make sense to first see a drop in quality before things improve.
AI firm code are likely a different category though.
Such issues are pretty common among people I talk to at least. I hear comments about it pretty often, both when it comes to ChatGPT and Claude.
For myself, with ChatGPT for example, it regularly gets extremely slow if there is a lot of text in a single conversation. Especially if I try to scroll up.
Some pages have been strangely just broken for a while now as well. Usage analytics just renders lots of these duplicate "Usage history" components, where the data just never loads: https://imgur.com/a/vpcQIiw.png
I can totally imagine a scenario where some agent built it, another tested and approved it, and nobody at OpenAI even looked at it once or knows that it's like this.
Not the one you asked but I genuinely think it’s possible that a part of the answer is the psychological effect of getting used to models performing well and picking up more if they fail, but also … If you have a product that works well for 95% of software engineering tasks (for the sake of this argument), with a large number of users there will inevitably be some poor schmucks who get bad results multiple times in a row. It would be highly unlikely if that _didn’t_ happen at all.
Now assuming you have millions of users, there will always be groups of thousands that experience this, and if those people go online to complain, it will look like an actual issue when it can just be explained by randomness and large numbers.
My best guess is random variation and rising expectations. I once saw someone do in depth manual testing of 3 different models and write up a whole report on their perceived strengths and weaknesses… and then discover that all 3 models were identical. Very good reminder of why the scientific method is important.
Serial testing over time is much less reliable than side by side testing, and even when I do side by side testing, I try to look at multiple attempts per prompt. Seeing multiple per prompt helps me realize how much intrinsic variation there is. My brain always wants to see patterns even when there isn’t enough data to prove them.
I'm realizing I am a minority, but my IDE is becoming MORE valuable to me, not less, with LLMs.
The volume of code is only increasing. Problems that used to be solved with a few lines (or even removing lines) are now solved with changes across 10 different files.
My IDE is invaluable for me when trying to navigate this stuff. I am completely convinced that people reviewing LLM code only via PR diffs are consistently missing massive systemic issues. You *need* to review changes in context of the surrounding code.
And doing this without an IDE is just making your life harder for no reason.
JetBrainer here. I can attest that you are a primary target for Air for IDEs.
We have put a lot of thought into how we can add value for the workflow you just described. Here's the way I think about it: FSD for a car. Can you imagine trying to sell a car without a steering wheel? I still want to code, I still want the deterministic side, even in cases where I'm letting the machine do the work.
I would really like Air to be closer to an agent / terminal multiplexer like CMUX / WMUX: I would for example like to be able to basically type "claude" or "codex" to launch my harness with the CLI args I need. The fact this requires dedicated API keys from the LLM providers makes it a no go at both home and work.
I wanted to try Air, as it seems to have interesting ideas for working collaboratively with LLMs.
But the choice of AI providers was very limited, and I found no ways to add the ones I use (the only options for providers were Anthropic, OpenAI and Google).
Unless you are blindly vibe coding everything without an IDE to help navigate what is being generated to review the output, working would be hellish.
And if you are blindly vibe coding everything, I am just glad I don't have to work with you. While LLMs mostly speed me up, they make serious mistakes often enough that a review is necessary.
So you're using the IDE for review? The main features you need are around simple things like reading files, tracking git changes, things like that?
That's a small subset of the functionality you would need if you're actually authoring code. So the IDE maybe be more valuable, but the feature set you need is actually much smaller and can probably be redesigned given your narrow workflow.
Sure, but my IDE does these simple things very well.
Just some important things off the top of my head that I use daily:
* Instantly jumping between definitions and usages
* Folding and unfolding code blocks
* Quick overview of structures in files
* Database access, so I can browse the data and the code at the same time
Also, sometimes I just want to quickly refactor/rename some variable across all its usages or whatever. It's just quicker to do it in the IDE with a simple keyboard shortcut, rather than prompting an agent "rename oldName to newName".
I'd agree with this. I've been working with Claude on an app. I wanted to increase the size of one element in the UI. I could have asked Claude to do it, but I wanted to understand the application, as well as it kinda seemed like a waste of time and tokens.
reply