> If you can direct AI and produce more value per $/time spent then a non software engineer directing AI to develop software then you still can produce a lot of value.
Even if true (which I'm leaning towards that it is), when will companies realize this?
We could be in for a decade long software career recession until the effects of poorly managed and written software trickles down to lost customers and company bankruptcies. Bad decisions in large companies take a very long time to produce clearly visible effects.
> As you develop your skills with AI, you now have the ability to develop your own software products.
I think this is a dead end. We're quickly approaching a state where software is no longer a product that can be sold, but rather a tool used by someone selling a service.
> We could be in for a decade long software career recession until the effects of poorly managed and written software trickles down to lost customers and company bankruptcies. Bad decisions in large companies take a very long time to produce clearly visible effects.
this does not mean software jobs would come back, you'd just end up in the amazon situation where today you use word made by JOXHSWW, then there are too many bugs and you buy another word made by ANSJJGG and so on.
Some B2B software will continue, but B2C can just become a single person fly by night with some models, or you can end up doing it yourself. A couple of years ago a model one-shotting a game of pacman was sci-fi, now it's just another benchmark, maybe the next model benchmarks will be one-shot photoshop or word or whatever. Either way today's C is more interested in doomscrolling tiktok on their phone than to booting up a computer to do some photo editing anyways.
As the article says, "taste" is still important in steering a model, but for how long? and how many people with taste do you really need compared to a few years ago where you needed people with taste AND a lot of people to execute.
It feels AI is just the next step of outsourcing, writing specs in the evening then magically the next morning software appears which might or might not work the way you expected, and the cycle repeated, only today the cycle is in your terminal and takes minutes vs hours/days
Completely agree. In my experience we're being more productive than ever but our projects are drowning in on the surface completely valid but on the larger scale meaningless changes. The bugs fixed were real, the features added are in some ways useful, but they really do not move the needle in any particular direction.
Isn't this a case of missing the trees for the forest though? The human brain is not an LLM, and an LLM is not intelligent in the same way as a human brain.
However, an LLM is a prediction machine, prediction IS at the very least one (or the most fundamental) element of intelligence. The brain most surely contains at least some kind of simulacrum of a prediction machine. How that prediction machine is used or wrapped is another matter.
If I said to you: "Blue blue blue, the color of my car is red", would you have absolute confidence in your prediction that my car is red? Or would the way I phrased that sentence make you slightly uncertain, and wonder if there's some miscommunication going on here?
Intelligence is compression, compression requires subtraction, and for some reason LLMs are not good at subtracting. To create a coherent model you kinda have to subtract correlations until only the essential parts are still there.
What I don't understand is why LLMs haven't been able to do this yet, if it's the harness or some orchestration layer above the LLM that is needed. Because fundamentally if you can identify correlations then it's just another small step to prioritize and remove lower value or irrelevant correlations.
I wonder if what's needed is to introduce subtraction tokens in some sense, and in post-training reward the model on that.
Intelligence is compression? What do you mean? Intuitively that doesn't seem right.
>What I don't understand is why LLMs haven't been able to do this yet
LLMs are just trained on what humans have said. Why is it surprising that it's still not possible to reconstruct the intelligence that wrote all that by working backwards? Think of your own work experience. When you look at a piece of code, say, are you always able to discern why the person did what they did, just from the code, with no additional context?
I guess they mean that intelligence is being able to hold models (compressed versions of reality) internally and use them to make predictions with a probability better than chance. That last part is the definition of information.
I find that highly questionable as a general description of what intelligence does. That's more like a description of a general knowledge base. When I think of someone intelligent, I think of someone who's able to draw unexpected connections between seemingly unrelated facts. In the broadest possible terms, I'd call it the ability to make abstractions and analogies. This is not just compression, but the ability to mentally operate on webs of meaning.
Unexpected connections between seemingly unrelated "models" :)
Is a fact stored on your brain like digits on a harddrive? No, it's a pathway that lights up and branches when information enters it. It is dynamic, a compressed form you could say, right? The model holds information, but not all information, but enough to be useful (in decision making).
Arguably it's the same, but the model is probably a "compressed" version of the whole fact that took place in reality.
And you can entertain the models internally and sharpen them. Alone or with others.
Doing those things also contributes to compression. I do recommend reading up on it, it's perhaps a little overstated for what people intuitively consider the two concepts but it's been quite well explored and has held up pretty well in practice.
There were experiments that zipped music pieces, I think, and then classified the compressed files by similarity. They got rather interesting resuls. But I do not remember much details. It was, I believe, about 20 years ago.
That sounds like a really cool project, I will def give it a look.
I still think this has much more to do with the structure of language than the abstract conception I have of intelligence, and I would be interested in having conversation w/ someone for whom the opposite is true.
bzip is not very intelligent, true, but it does develop some model of its input. It's not like there's a linear relationship between between compression ratio and IQ or anything.
I'm not arguing that compression algorithms don't produce models of their inputs, some even use neural nets or other stochastic predictors with correction terms etc.
This doesn't address the only part I really commented on, which is the connection between intelligence and compression.
Creating the model takes intelligence, but running it doesn’t. I think the point everybody’s revolving around is that the transformer model is an absurdly inefficient and low-fidelity approximation of a system that acts, observes consequences, and incorporates that feedback going forward.
The issue isn’t really harness vs. no harness. IMO it’s about the lack of an internally generated sense of what to attend to. Yes, the KV cache accumulates state and its “attention” (if you can even call it that) changes with context. We’ve even managed to /kinda/ close the loop with agentic tool calling and ‘memory’ systems, but these just close the loop at the level of behavior rather than disposition. All agentic harnesses do is make an LLM responsive to the consequences of its actions without changing the tendencies by which it determines what to retain or avoid.
The ghost you can’t escape from at this point is the origin of that relevance. Where does the pull toward one thing mattering over another actually come from? If you ran Fable 5 on a Turing machine and rewound the tape to the exact same state with the exact same input (incl. PRNG seed), it would spit out the same output every time.
Everyone’s trying to outrun this problem by training more often or increasing model sizes. But all this does is inform your model, from the outside(!), what constitutes a better state. The thing that’s actually doing the determining remains unchanged. Congratulations, you’ve scaled the transition function and tape of your Turing machine until it requires every watt generated by ERCOT, and it still cannot, for the life of it, tell you why it should give a shit.
A trained model generating output from weights, a seed, and some context effectively has next-state that’s a total function of those three things. Whatever behavior appears as ‘selecting what is relevant’ is, underneath, just a transition rule executing, no matter how sophisticated or creative the output looks. It can be fully accounted for by what was fixed before it started executing. Which means whatever criterion it uses for determining what matters was inherited from a structure that was already in place before it encountered the situation.
No amount of pruning or post-training can fix this. These approaches just replace one externally supplied criterion with another. For a system to be truly adaptable, there would have to be some criterion by which it treats one possible change as preferable to another, and that criterion itself would have to come from... somewhere. You can even change your conception of ‘improvement’ (e.g. parameter count, harnesses, self-modification, hell, even its ability to spit out shitty best-selling romance novels onto Amazon) and you still haven’t explained where the normative distinction comes from. Every layer of this problem has its root in a preference that was supplied from somewhere else.
I genuinely don’t know if this issue bottoms out anywhere, at least for the way we currently build these systems. Perhaps the solution is still computable, maybe? Who knows what that would even look like. But I’m fairly confident that it isn’t a bigger tape. I hope nobody solves this in the near future because, well, I’d like to have a job...
You're so close... And where is the magic "uncomputable spark" located inside of you? If you say analog thermodynamic noise - then ok, if we use true thermodynamic RNG for LLM activation function, will that meet the criteria? But what if super determinism is the law of the land? Then nobody is anything but computable from priors...
Ehm it's literally every cell in my body. We can't simulate a living cell, we're orders of magnitude off before we can do that, the onus is on you to make an argument that it is in fact, remotely similar to what you describe as "computation".
It's not some tiny "uncomputable spark" you need to look for, most of it is entirely uncomputable.
Where is the computable part in me that is doing all this thinking and being a person? Where is that "tiny spark", point me at it :)
Completely agree. AI is very impressive in many ways but there is something deeply wrong that is hard to put into words. The output is probable but never true, if that makes sense.
I think this is also the mechanism behind why AI generated videos and images are so captivating at first. I remember when Midjourney first launched and it was hours and hours of a brain-melting "Wooooooow". But once you get used to it and start to identify the patterns the brain quickly labels most AI-generated content as blank space.
If the image or text wasn't created by a human, then there was no intent behind the content, there is no message or novel information conveyed, and it reads as noise.
Yeah AI generated content hints that there is a whole world behind it, the way that an image pre-AI was a clue that there was a rich 3D space that corresponded to the image.
It seems our brains are adapting to that and recognizing "actually the signal behind this message is quite sparse" even when presented with rich imagery.
> If the image or text wasn't created by a human, then there was no intent behind the content, there is no message or novel information conveyed, and it reads as noise.
If I were to push you a bit on this, when is it not true?
Let's not like at AI specifically, but can you think of other examples? Like for me, I think of: the creation of earth itself, or stars, or even DNA.
> But once you get used to it and start to identify the patterns the brain quickly labels most AI-generated content as blank space.
I guess the majority of people do low-effort generation that doesn't perturb a default style of a network enough, so it stays blatantly noticeable. The percentage of "super-recognizers" who notice almost all AI-generated images is around 1-2%. It could be that you are one of them, of course.
I think this is the confirmation bias trap a lot of people fall into; higher quality, hard to detect AI is already ubiquitous but because it's hard to detect people just don't clock it.
"I can accurately detect 100% of AI generated images that I recognise as being AI", if you will.
I think the last one is by far the biggest reason for why this happens way less in humans. In every conversation, almost for every sentence, people gauge the response their words have on whoever is listening. If something didn't land as expected (frown, confusion, unpredicted response) you unconsciously adapt and try something slightly different.
It's immediately obvious by the fact that you're clearly using way different ways of speaking (tone, speed, vocabulary) when you're speaking with a friend, versus your parents, your colleagues, people you don't know, children, etc.
Maybe, or maybe you are overly pattern matching on what you what to be true?
I don't think anyone would mind having a competent manager or PM in a standup, someone that is actually contributing towards finding solutions and ways to move forward.
yeah, I would say half of the program managers I ever interacted with had their role justified, the other half would be let go during the next layoff....along with at least half the competent ones..
Isn't this only a transitionary problem though? Right now, during the transition there is no incentive for humans to write anything, it will get drowned in AI slop.
As time goes on, more and more people will recognize this problem and we'll develop new ways of measuring information quality and trustworthiness. Nothing about this problem is fundamental, it's just that we're in the middle of a very chaotic transition.
The end goal though (in my understanding) has never been for an LLM to regurgitate knowledge it ingested during its training. The end goal is to use the patterns and correlations found in internet data to generate an emergent prediction and problem-solving machine.
Whether that's possible is something we'll discover, but no one is throwing billions on AI companies for the hope of them building a giant natural language queryable internet information repository.
For well-scoped tasks I wouldn't say the code is bad, not brilliant for sure, but definitively good enough.
For a lot of problems "quick and good enough" is all that is required. I've used it a lot for managing my Home Assistant setup. Has saved me countless of hours.
In principle I could've done it myself, but I never would have, the time investment required to learn it wouldn't have been worth the value I get from it.
Even if true (which I'm leaning towards that it is), when will companies realize this?
We could be in for a decade long software career recession until the effects of poorly managed and written software trickles down to lost customers and company bankruptcies. Bad decisions in large companies take a very long time to produce clearly visible effects.
> As you develop your skills with AI, you now have the ability to develop your own software products.
I think this is a dead end. We're quickly approaching a state where software is no longer a product that can be sold, but rather a tool used by someone selling a service.
reply