> Knowledge and reasoning are inextricably interwoven in the weights of the neural network—there is no independent, explicitly represented set of beliefs.
I'm not sure I understand this. Do we have evidence humans have an independent set of beliefs not shaped by knowledge and reasoning? If so, where do these come from?
I'm especially confused about a prior statement as a scientist:
> Three shortcomings prevent what chatbots do from qualifying as reasoning (in a way that a scientist might recognize).
How does a set of beliefs help with reasoning?
> Third, while the chains of thought chatbots produce look like deliberation, research has demonstrated that the bots often concoct them after the fact, reaching an answer by one route but reporting another.
Philosophers like Kant, Descartes and Locke wanted to draw a distinction between a priori and a posteriori beliefs (or knowledge). I believe AI will be very much bound by these discussions. Some knowledge can be had purely by reason. Some other knowledge will require observations.
Consider a calculator, a machine capable of valid transformations wrt the semantics of arithmetic.
Does a calculator reason its way to 2 + 2 = 4? No, it merely transforms.
An LLM is simply a much more complex version of this; except that its complexity hides a lot of what is otherwise reducible to a transformation of semantic, conceptual, and inferential relations. There are lots of NLP models prior to the introduction of modern "AI" that would make these representations more clear.
However I don't believe any of the aforementioned philosophers would ever consider reason itself as a transformation, as Kant especially spent a lot of time differentiating apprehension from transition.
I’m not so sure. While I completely agree on pure transformation, I continue to believe that the most concerning aspect of neural networks is that they really strengthen the argument that consciousness is the illusion.
The interplay between deductive frameworks and inductive frameworks is still always buoyed by the black swan problem… which is a the heart of Hume’s solipsism. Kant’s pure reason never considered instinctual logical gates that come prepackaged in the mind. That seems frighteningly close to a prediction algorithm, no matter how much we wrap it in a concept of understanding. In the end, we’re just building a model with a minimal error rate.
Western tradition has firmly put pure reasoning in a cage. It serves its purpose only as a source of hypotheses, conjecture and thought experiments. Abrahamic religions, while still not accepted sensu proprio by many, are much less mystical than the Eastern Religions. You can see the difference in the unusual creative process of Ramanujan.
Which tradition are we to follow then whilst training the AI models? Pushed beyond their limits today's models tend to hallucinate and cannot be accepted sensu proprio. Philosophers will call it speculative reasoning. And yet, some speculation can be useful in breaking new grounds. Anyway, I have no answers. Just musings.
> I continue to believe that the most concerning aspect of neural networks is that they really strengthen the argument that consciousness is the illusion.
I think it’s not a question of it existing, but to what extent does conscious can control the organism (as opposed to being a passenger). There’s some intriguing research on the topic.
> I continue to believe that the most concerning aspect of neural networks is that they really strengthen the argument that consciousness is the illusion.
Based on what? Just because you can model the weather using math doesn't imply at all that the weather is the product of a mathematical model itself. I don't see how reasoning is different.
We deliberately construct LLMs so that they approximate certain relations typical of reasoning. Suppose they become extraordinarily successful at doing so. What exactly have we demonstrated? That these effects can be artificially reproduced through such a mechanism. But how would it follow that the thing being reproduced must itself be nothing more than that mechanism?
Based on the face that we have zero explanation for consciousness. Based on the face that biologically neural networks are exactly a kind of neural network.
My point isn’t that we’re able to explain consciousness as an illusion. It’s that we’re able to explain consciousness-like behavior presumably without consciousness, and the infrastructure look uncomfortably like we do.
What does consciousness have to do with reasoning? You're conflating two different things.
Consciousness is the what-it's-like of sensation you have as an organism from your sensory modalities and having a body. For human, that's colors, sounds, tastes, smells, feels, feeling of balance and what not that happen in perception, hallucination, memory, imagination, inner dialog and dreams along with emotions. Each sentient organism has their own sensations and emotions making up subjective experiences that nobody else has access to except inference from behavior and language (for humans).
That's not reasoning (just the having sensations/emotions). Reasoning is an adjacent activity that can be conscious but also unconscious. And it's hard to say that colors, sounds, emotions are illusions. What does that mean? We do experience them. There is something it's like to see color which is different from what it's like to feel pain or to be in love. And none of those (presumably) are what it's like when a bat experiences echolocation (there are many examples from the animal kingdom if one takes issue with echolocation).
> Knowledge and reasoning are inextricably interwoven in the weights of the neural network—there is no independent, explicitly represented set of beliefs.
I think an analogy would be helpful. As an LLM, reasoning through text, you wouldn't 'know' the idea of a man separately from, posterior to, the words man in say, French and English. As a human, you WOULD know the idea of man, and you would mean that idea when you say the French or English words for man.
For an LLM, although its approximation of knowledge would lead it to claim it knows they're the same thing, there would be differences in its weights that influence its usage of both the French and English words for man, and which may lead it to conclusions in one language it wouldn't reach in another. Because its knowledge/reasoning is interwoven to its knowledge, not prior to it.
It’s not, because it’s not evidence for the argument in the article.
> As a human, you WOULD know the idea of man
You’re making a claim about the fundamental mechanics of human biology. I don’t take these at face value. For one, language (in a very broad sense) put a major selective pressure on our species. So, to test your hypothesis, we’d need to find people who do not use any language and somehow test they have an abstract idea of a man.
To put it otherwise, I don’t know whether our reasoning can be decoupled from speech or language. There’re some bits and pieces here and there, like deaf people being at higher risk of neurodegeneration.
And let’s not forget there’re claims that AI models develop internal representations of various concepts. Anthropic made a very interesting claim, but I’m not an ML expert so can’t judge whether it’s sound.
In a human brain it can reason regardless of what you personally know. In an LLM, the knowledge and the structure are the same thing. It doesn’t have separate knowledge parts and processing parts, it’s all one network. If you delete the part about cats, it can’t think at all anymore. Whereas a human could lose all their memories and still be capable of thinking.
How can you know that, when every human who reasons knows a lot? We spend years in a pre-rational state, and by the time we can reason reliably we have ingested a huge amount of evidence about the world. Memories are just a snippet of what you learn by observation; you don't have a memory of gravity but you do have an internalised model of gravity derived from what you learnt as a baby.
I don’t know why you guys keep rehashing what the author said. I don’t really have a reading comprehension problem. I have a problem with someone taking it a face value. See above for my other responses.
I think a better argument for them not reasoning is that they are incapable -- seemingly, for now, that I know of -- of independently starting a reasoning task.
They do not ask whether or not they dream of electric sheep. Unless prompted to do so.
That's part of human reasoning, being prompted, but so is independent thought.
So at best they are partially reasoning, unless we philosophically argue these are two separate things. And frankly, shrug, I'm not a philosopher.
In the early reasoning models the chain of thought would often drift off into other topics, almost like a daydreaming. This behavior is undesirable so that's been RL'd away, but I do think this is something we could evoke from models. It's just difficult to contend with from an eval perspective.
This is a good point and we would have to understand whether this is noise in the machine or independent reasoning. I haven no idea how to measure that.
If somehow independent reasoning then we're getting very close to something alive that would need rights.
In my view.
And I have yet to have an experience with an LLM where I came away thinking -- this thing is alive and me using it this way is unethical. I've had that experience with animals, even people. Never -- yet -- an LLM.
> they are incapable [...] of independently starting a reasoning task.
This is a simple and intentional design choice though. Biological lifeforms are always on and always receiving sensory input. LLMs don't functionally exist out side of when we decide to run them. An always on agent with a looping prompt of "if you aren't doing anything else, ruminate" overcomes this limitation.
In fact, if your brain were removed from your body, and you were locked in a room that was completely empty and precisely calibrated to your brain's ambient temperature, and you were then ordered on your life to not ruminate. Well -- I would posit that you wouldn't last very long.
There's something else going on with human cognition -- and reasoning by extension -- that LLMs are, to date, not replicating.
With the big caveat of AFAIK. I'm not in one of the labs close to this stuff.
> Not really. Noone has to prompt you to ruminate.
Why do the internal mechanics need to mirror how it works in humans?
I have agents that receive sensory input (video/audio) which run continuously processing it, receiving events as they occur. When something strange happens they notice and take action (i.e. there's an unrecognized person in the living room -> ask who they are.)
Presumably you've seen the agents that play video games - build things to progress in Minecraft, plan routes, avoid adversaries, etc.
Why doesn't that count?
I feel like you're failing to consider anything outside the chatbot UI.
EDIT: The first morning my agents had access to an outside video stream they commented on the sunrise. While I was sleeping through it.
> they are incapable ... of independently starting a reasoning task.
I have provided several examples of them independently starting reasoning tasks.
... and I'm wondering why you seem to be entirely unaware of such examples. The most likely explanation being you are only aware of the chat interface.
... and I have absolutely no idea of what you mean by "... and feel".
I think you’re on the right path. My first thought was whether lack of constant sensory input is an issue. We’re always feeling something (save for sensory deprivation chambers, but even then…)
Or put it other way, lack of feedback loops between the “brain” and the environment.
> while the chains of thought chatbots produce look like deliberation, research has demonstrated that the bots often concoct them after the fact, reaching an answer by one route but reporting another.
Doesn’t research show humans often do this too? There’s a pretty famous paper from the 70s about that [1], and lots of subsequent evidence. We also have choice blindness [2], we confabulate reasons [3], and we even change our choices (sometimes negatively) after trying to introspect [4].
This is the double edged sword of calling it AI, of using terms like Temperature and Hallucinate and Thought.
Stop trying to compare either system to a human and look at it for what it is -
A prediction engine that runs fast enough to brute force problems.
In the case of alpha go its "innovation" was millions of games played against itself. It had bound parameters and strict win conditions.
In the case of LLM's you can deploy 1000's of agents to smash themselves against an idea. The whole hugging face attack is an example of this (1200 agents out of an unknown number chose that path).
There is the old saying about monkeys, typewriters and Shakespeare. Well we have better monkeys who basically follow a derivative of zipfs law (not actually), who use tokens not letters and their goal in many cases is testable (compile, unit, E2E).
>A prediction engine that runs fast enough to brute force problems.
The interesting thing is you think humans don't run in the same manner a lot, if not most of the time.
When there were very few humans on earth, development was very slow. If I sent you back 10,000 years ago you could catch up humanity 9,500 years or so with just the knowledge you've learned via memorization. So this idea that humans are pure reasoning machines, each one capable of great feats of logic just doesn't seem to hold true. Instead deep reasoning and insights came very slowly over time and as we built up technologies like writing and reading our abilities to exchange information increased over time. This lead to more people, which further brute forced the problems of humanity.
Dont be fooled. Reasoning does not happen in the prediction of the next token, it happens virtually in the text that is created.
The next token prediction is just "the hardware" following the underlying rules. Like the basic set of rules.. in a sense similar to how the "game of life" does not really contain gliders. Gliders are just a self stabilised system that arrises from the simple rules.
All these "AI can't" arguments seem to secretly rely on the assumption that human reasoning/sentience/whatnot is dependent on a soul (or any equivalent metaphysical entity). Everything else is handwaving.
I agree - many are. But there's another camp to watch - LeCun/etc don't believe in a soul but also don't believe that LLMs are capable, because they don't contain the right neural network.
I'd argue that they're still intuitively trying to keep intelligence in the gaps, as humans like to do. Or maybe they're mad they bet on the wrong horse... or maybe both. Hard to say.
> because they don't contain the right neural network.
I'm totally onboard with that as a plausible argument, but I think the architectural limitations with respect to consciousness are very deliberate rather than some lack of technology. We've invested huge sums of money and human effort to create tools with explicit goals that are completely at odds with consciousness or AGI. If we had invested similarly with the clear goal of creating something with agency, self-determination, neuroplasticity, etc. instead of controllability, repeatability, reliability, I think we would be there already.
Maybe. I'm just suspicious of the certainty of the people who are certain of it.
And... I remember (all of six years ago) back when language was considered the pinnacle of the human mind. Sure, animals might be smart, but they don't have language!
The moment LLMs appeared it suddenly took a back seat to physical navigation and child rearing.
Feels like more gap seeking to me. But time will tell.
This is trivially false, the text gets transformed into activations for the weights. If there is reasoning it's in the connection pattern of the weights.
The fact that the output produces one token at a time does not mean that the LLM's internal state is processing just the next token
When i look at the things said by LLMs i clearly see stuff that i would call reasoning.
Its reasoning does however have certain "bugs" that a humans reasoning would never have.
My guess is that much of those bugs appear cause the reasoning that a LLM does is not itself built on a self stabilised system. In humans you get coupling between levels of self stability which acts as a constraint, stabilising the system even more. The next level being predictive coding modelling the world. There is no next level in a LLM; they are trained as refiners in teacher forcing mode, a paradigm where self stabilisation is not a driving factor.
My pet theory is that there are different types of intelligence that have different pros and cons. Social/cultural, intuition, and structural.
Structural is like step by step reasoning or math or raw compute.
Intuition is statistical from repeated trial and error.
And social is leaning on the wisdom of the crowds. So like high latitude countries where they eat fish for breakfast and get better health outcomes.
So I believe that LLMs have stumbled upon a partial component of our social intelligence. Word distribution, ontologies, jargon, information theory (frequently used symbols should be short). We mutate the language that we speak to be useful to us based on the problems we face. To some extent being able to talk the talk means you can also walk the walk. At least partially.
It's kind of shocking how far they can get, but at the same time it's kind of a surprise how far they don't. The existence of agentic harnesses is sort of an admission of defeat.
While some might be fooled into thinking that they reason, everyone I've met isn't. As a software engineer I'm drowning in work. And if that's not an admission that this isn't a real intelligence then I don't know what is.
But ultimately it looks like we've got all the individual components sorted. The old school 70s era stuff has a lot of the structural intelligence covered. The data science era of statistical ML has the intuition. And LLMs have the intelligence from our culture.
Maybe there are more general or energy efficient or powerful or special purpose techniques out there. And maybe combining everything together requires some additional insight. Regardless it feels like moving forward to something better than our current AI landscape is plausible, albeit with a completely unknown level of effort.
I don't know what you would call it, but reasoning/thinking/whatever it is, is a way for LLMs to tighten their sampling space, while allowing for wide sampling to still happen. This is akin to people brainstorming ideas.
I know that sounds confusing, let me break down how I think about this.
1. LLMs don't pick the token that ends up being used. This is by design, if the LLM gives a wide choice, it can better adapt to real world scenarios. i.e. generalize.
2. Without reasoning, this means that the LLM either locks in on whatever the sampler picked. Or decides mid-sentence/response to correct itself. This is what used to happen before reasoning, still happens if you turn reasoning off.
3. With reasoning, the LLM can make as many mistakes as it wants and explore its sampling space. Then use its vast pattern matching capabilities to decide which parts of the reasoning make sense and which were idiot ideas.
4. Enabling reasoning makes it so LLMs are much more confident on the final response, and the logits should theoretically all be near 99% on a single token for every token, i.e. much closer to greedy decoding. It analyzed all the possible options and figured out the best outcome, so a stray sample doesn't cause the answer to go awry.
This is why reasoning traces are filled with "but wait". I don't know if those were added in organically or artificially in the RL training, but regardless they're a good way to let the LLM keep generating other options and explore it's sampling space to the fullest.
Note: I haven't tested any of this and it's just my theory, but I'm sure if you really wanna know you can have claude run some smoke tests :)
People should read the article instead of responding with what they believe to be clever quips. The article's author gives a very good argument for why what LLMs are doing in their chain of thought is not reasoning.
Yes, and keep in mind also the article author is chair of machine learning at University College London and was a core member of the AlphaGo team at DeepMind.
This said, just because it's an argument from authority doesn't necessarily mean it's a good argument. You'll have to read and break down the arguments and weight them against other arguments and the body of knowledge we have.
If argument from authority was valid in itself, then LeCun would have killed LLMs like 300 different times now, and yet keeps being wrong.
Although they don't mention anything, the authors website seems like they are tee'ing up for a new start-up. They present their future ambitions (merging two technologies they know well) and that they recently left deepmind, so it at least smells a lot like laying groundwork for a new start-up.
Law of headlines. Past the slightly inflammatory framing, the article actually makes a case for making LLM reasoning more rigorous. Which, you know what, is absolutely something you could train them on. Might be worth exploring if we can apply the rigor rigorously. You could have smaller models that converge at all on harder problems, and larger ones that converge faster.
And a magician making a coin "disappear" doesn't mean that magic is real. I see no way we can call it thinking without a goal (other than computing the next token).
This is the problem I have. People are built to be fooled by this stuff - to see something that's not there. Pareidolia but with language. It's such a compulsion that the people who build LLM's see that the same non thing there too.
When I stumble around trying to explain my thinking though I invariably get hit with the response, Sure, but if it's functionally identical to a coin being magically pulled out your ear, what's the difference if it is or if it isn't? The only response I have is that when the coin doesn't appear you're going to be putting yourself way further behind the starting line then if you thought from the get go that there's no such thing as magic.
> This is the problem I have. People are built to be fooled by this stuff - to see something that's not there. Pareidolia but with language.
I really like this metaphor, it hits home that just because you sense something is human-like doesn't make it so. We're wired to respond to those sorts of things.
> When I stumble around trying to explain my thinking
I had similar difficulties expressing my thoughts in an organized way. The video below from Richard Sutton (the father of Reinforcement Learning [which LLMs use extensively in training]), helped bring order and firm up some of my thoughts and theories.
>I invariably get hit with the response, Sure, but if it's functionally identical to a coin being magically pulled out your ear, what's the difference if it is or if it isn't?
Not to spoil the video, but it isn't at all functionally equivalent. LLMs are an imperfect and somewhat randomized simulation of what the LLM thinks an average person might say in response to a question. Which is another way of saying, most people will on average get worse answers than if they worked on a problem themselves (but they will get that worse answer from the LLM comparatively quickly). Practical people respond to this positioning fairly well, and for others it still lands a bit since no one wants to think they're below average. That said, I'm still working on my positioning a bit as well!
I believe there's a confusion in this thread between what an LLM _does_ and what an LLM _is_. What is does is output a next token. What it is, is a universal function approximator ([1], i.e. a neural net).
With back probation in the neural net, there could be a full state machine being approximated inside the weight. And a state machine is the exact step-wise reasoning the author claims it needs.
You are almost there in terms of getting their point, so I will explain.
Birds existed since forever ago in nature, and they fly by flapping their wings. Then planes got invented, and they fly using a very different mechanism (that doesn't involve flapping wings).
The point made by the grandparent comment: saying "LLMs don't actually reason, because the underlying mechanism they use is different from how humans reason" feels about the same as "planes don't actually fly, because the underlying mechanism they use is different from how birds fly".
Comparisons to how humans reason beg the question: is the way humans do it the only way?
Lots if these arguments are similar to birds saying "Jets don't flap their wings so they aren't even flying."
The arguments about reasoning are even shakier because they usually rely on totally unproven assertions about human reasoning. At least we know birds flap their wings.
Copying biological systems isn’t always the best way to build machines. The article compares AlphaGo’s policy network and value network to system 1 and system 2 thinking in humans.
This comparison between fly and intelligence is worthy of the most vulgar bar talk
> it can't have superhuman reasoning
no they don't, we still die of cancer, there's no global deployed autonomous driving and food production driven by super intelligent ais and I'm not walking on mars thanks to gravitational elevators
I disagree. I haven't finished the article yet, but it smacks of arguing for mechanism over result.
If by some mechanism other than what a gatekeeper would call 'reasoning', a machine produces outputs that approach indistinguishable from 'well reasoned', the argument that it didn't get there by reasoning is, well, not useful at the very least.
Your last sentence is identical to your first sentence in your other comment. How can we be sure that you are reasoning and not just a stochastic parrot?
Humans have an inherent flaw in that our brains are wired to see intelligence and reasoning where there is none. Our brains fill in data that simply isn’t there.
Folks see Jesus in burnt toast. Monet was a master of exploiting this where what’s really just blotches of color our brains fill into beautifully detailed images.
Our experience with LLMs is no different. Folks believe there is some deeper intelligence there but it’s all still just 1s and 0s on a computer chip. We’re interpreting things happening that simply are not happening.
You either have to accept that the brain can be described with math (like everything else we have ever known in the universe), or that there is a supernatural phenomenon that exists in the brain.
This is an inescapable conclusion that boils down to "Do you believe magic is real or not?"
Magic is real and you can have your unique special human intelligence.
Magic is not real, and the brain is just another computer crunching numbers.
I think it's been pretty well demonstrated by neuroscientists that the brain is not a binary computer. That doesn't mean that those scientists erred on behalf of supernatural religion or anything like that.
Nobody is actually trying to do that, because it's too hard, they don't know enough about how the brain works, and/or they can't compete on efficiency; pick whichever reasons you like, or add more, but don't pretend that AI researchers have figured out how to simulate a brain.
Not really. He said no one who knows anything about the brain thinks it's a binary system.
AFAIK, that is true.
Whether it's possible to replicate what happens in the brain on a binary system or not is a separate thing entirely. It's also not what LLMs are attempting to do.
You're reading him too narrowly. He's answering the question of whether the brain can be modeled (math or magic), and arguing against it. Or maybe he's arguing for a middle ground. Hard to say exactly, he's not very clear.
Except the brain can be modeled with arbitrary precision - so he's got an uphill battle in either case.
TLDR: Yes, he literally said "binary computer" but from the context it's clear he means more than that.
Because the metaphysical is definitionally outside the realm of science and rationality, and every bit of scientific and technological progress humankind has made has come from discarding superstition and approaching the world as a physical system.
If the brain isn't just a physical system, then we may as well give up. There's certainly no point discussing it, as any assertions will be untestable and one persons elaborate and well thought out theory will be just as valid and predictive as the next persons "consciousness is created by invisible purple unicorns" theory.
So if we are going to discuss it, we should start from the assumption that there is no magic or witchcraft or religion involved.
> If the brain isn't just a physical system, then we may as well give up
Forgive my obtuseness, but if it turns out that the brain is more than just a physical system, what are we giving up? On science as a whole?
Isn’t science predicated on reliably predicting reality? Do we know how to describe reality accurately?
I asked below as well, but how does the observer effect work as a physical system? I get we don’t know so you can’t really answer, but if we can’t accurately describe reality then our science is woefully incomplete! Given that incompleteness I really don’t understand how we can rule out “invisible purple unicorns”?
The issue is that we have yet to prove or find evidence for the existence of anything but the physical. When we do find a new phenomenom in one of these ways, it becomes a part of physics. What would it mean to concretely discover a non-physical phenomenon? It would have to genuinely be magic of some sort, and defy rationality and description, to not simply be a new part of physics. (And if it defies rationality and description, it doesn't advance us scientifically, except to let us know we must abandon science in that domain.)
> I asked below as well, but how does the observer effect work as a physical system?
Just to be sure - the observer effect doesn't refer to a human observer, but rather to any measurement of a quantum system.
It is unsatisfying that quantum mechanics posits that particle states are fundamentally probabilistic. It's possible that in some sense there is an underlying order such as string theory that is impossible to test or verify. In that case, it would make sense to consider string theory as metaphysics. A fun exercise, but not scientific and not useful for understanding the world. By definition it would not be helpful in predicting or modeling the world, otherwise that would serve as a way of testing the theory.
> It is unsatisfying that quantum mechanics posits that particle states are fundamentally probabilistic.
You can alleviate this mostly by accepting the MWI.
Under the MWI, all the possibilities do occur, and the probability manifests because when you split into an infinitude of yous, each of them is just one you, and has no way of predicting which you it will be on the other side.
So the MWI makes the wave equation deterministic when viewed from the outside. The trouble is that we are inside it.
It turns out that God does throw dice - it's just that all the combinations come up at the same time.
That's fine. Just understand that modern science will likely explain this by a malfunction of your senses - because no evidence of paranormal activity has ever been shown.
If you want to argue outside of science you should state it up front - especially on a site like this.
It shows that it doesn't matter. There is nothing a brain can compute that a base 2 system cannot.
People can drill really really hard on the fact that the brain doesn't function with purely 2 states, but it gets you nowhere. It's the same illusion as "pure analog music sources are "better" than digital sources". They're not, and it's a totally immaterial topic when discussing how the music sounds (audiophiles, come at me). Either system can produce the same sound, indiscernibly, even to the fanciest test equipment.
The brain also cannot escape that it's digital clone mirrors it's inputs and outputs to an arbitrary point of perfection.
> You either have to accept that the brain can be described with math (like everything else we have ever known in the universe)
Oh the irony of you talking about believing in magic. What do you base this wild claim on? Can you cite a single known physicist or mathematician that agrees?
> And so in its actual procedure physics studies not these inscrutable qualities, but pointer-readings which we can observe. The readings, it is true, reflect the fluctuations of the world-qualities; but our exact knowledge is of the readings, not of the qualities. The former have as much resemblance to the latter as a telephone number has to a subscriber.
I'd be delighted if rather than an old dead guy quote, you could provide an actual example of something math comes up short in describing, for lack of mathematical ability of course, not lack of human knowledge or practicality.
So yes, you can cite no serious scientists. And you don't understand the words of those, be they current ones or any of the heroes of either field, when they explain why that is.
You don't even need to believe them, just think about how you would ever be sure you know objective reality fully. Actually do it.
When I was a little kid, my go to thought experiment was imagining atoms as balls that we can't crack open that are filled with sand, and that even if we could find the formulas that describe their movements perfectly, we could never know if that's how atoms move, or if there's something inside them (the sand). Really dumb, but I was like 9 yo and knew nothing, never heard of Gödel or quantum mechanics (what's your excuse?). But I still could understand, not intuitively, but by actually thinking (not just talking) about it, that you can never be 100% sure from inside. Even if you found out everything, and your model of the world matches it perfectly, you could never be sure that it is so. And it turns out this is true and an old discovery. If you could disprove it, you would be famous for millenia, maybe forever. Bluffing on HN won't get you there.
Maybe the brain is just another computer crunching numbers, but with real epistemic representations, rather than just predicting a token, based on a corpus of syntax.
Does it matter if "intelligence and reasoning" are there if what they produce is indistinguishable (or outmatches) what a human could produce?
You only have your own experience to judge that you are even reasoning. You assume others have similar experience and so reason like you because they are able to do all the things that you do - and they look human like you. We can't prove there's a there there in other humans. What happens when the robots are doing everything humans do? What happens when they tell us they are reasoning, when they say they have an internal experience? Maybe there's nothing there, but you can't prove it. Moreover, it's likely they'll be able to affect the world and you in most of the same ways as humans can, whether there's a there there or not.
I can give Monet a pass as he probably didn't fully understand WHY it happened and it was just art, but the way tech companies exploit our brains (algorithmic dopamine hits, LLMs, etc) is pure insidiousness. The fact they act like victims when the backlashes come is what really grinds me.
This just reads so anthropocentric to me. Humans are wired to see intelligence only where intelligence is human-like. We see autonomous action-response and planning as the keys to intelligence. I would expect that, dear primate based Homo sapiens.
We are experiencing non-humanoid intelligence without AGI. That is awesome. And we don’t have a clue how to protect ourself from AGI.
Likewise, we intelligent primates have this great system of coordination called market economics that lets us destroy our home planet with our eyes open. That’s what we call intelligence!
(Not 100% personal opinion and deliberately inflated from the I’ve been thinking about.)
I'm curious about some of the claims in TFA and how it lines up with the research we've seen so far. The author of TFA is clearly very experienced in the field, so is there disagreement on what this research means? Hoping experts can chime in:
> First, these models typically maintain no explicit, persistent, and inspectable epistemic state... there is no independent, explicitly represented set of beliefs.
We cannot decipher it, but Mechanistic Interpretability research does show that models are applying and manipulating abstract concepts and relationships encoded in the weights to derive their responses. We can even identify and manipulate those weights, see e.g. Golden Gate Claude. I assume these are the "independent set of beliefs" and the only reason they are not "explicitly represented" is that the representation is too complicated for us to decode.
In fact, this is also how we know they "concoct" chains of thought, because their reasoning traces do not always align with what is going on in their weights (i.e. the "concepts" that were activated during inference.) Lookup chain of thought faithfulness research, something TFA directly cites.
As another comment (https://news.ycombinator.com/item?id=49934166) points out, there is very robustly replicated evidence that humans do something similar (lookup "post-hoc rationalization.") So it is extremely fascinating that LLMs do the same thing!
Can't help but wonder if that's an entirely unrelated though similar-looking phenomenon, or an emergent property of intelligence, or something transmitted subliminally via training on the data our brains we produced...
I relate a lot to what they do. Find words, alignment and suss out follow ups, ons, and outs to the next reasonable conclusion.
Then use that context to bootstrap the next because if you build a powerful conclusion than can reverse itself into its evidentiary context, then every next context step can update its priors.
And so on the turtles flow where like an LLM, THE start of the context disappears over the horizon, but as long as im contexting in disinterested chunks of equal quality, then its not a problem.
But while internal tobeach context you can find reason, as a requisite building block like falling tetris pieces, the whole isnt the sum of its parts.
This is an ad for the new startup by the author, who will now focus on reasoning models. His insight seems to be that LLMs should make their assumptions explicit first, then map out a search space and provide reasons for why they should take one path or the other.
I think it's an interesting approach but the overall discussion about reasoning is really pedantic. What the author is describing here is one approach out of many, and in my opinion it doesn't cover what humans colloquially think of when they hear reasoning (while the output from a chain of thought sometimes does).
I disagree. They do reason during the reenforcement learning stage. They don't reason at inference. A good metaphor is that useful output are like nuggets that exist after reenforcement learning which need to mined to be, in LLM talk, "surfaced." Without supervised fine tuning, the reasoning models will add weight to tokens, words and phrases like "verify" and "check work" which will cause it to follow those verifying tokens with reasoning tokens that do just that, verify.
"Intelligence measures an agent's ability to achieve goals in a wide range of environments"
Interestingly in 2019 this was the author's take. He now appears to be confusing/conflating between LLMs and agents in a way that helps argue his case about "System 1", but his prior view seems more metaphysically robust.
From Jacobian. They developed a tool called a Jacobian Lens, or J-Lens (itself a descendant of an earlier simpler tool called a Logit Lens) to examine what was going on inside an LLM, and named the space they found with it a J-Space.
Jacobians are essentially derivatives but for matrices.
Brute force is powerful yes. And another way to look at math problems is that humans solved a huge lot of very hard math problems already but didn't solve all of them.
Another thing humans did is invent the telescope, the microscope, the transistor, antibiotics, the computer, AI, discovered how to send satellites in space and how to do heart-transplant etc.
I'd say there's still some way to go for AI before we declare humans dumb because they "failed for decades" at solving a few math problems.
An LLM is just a brain in a vat.
Our brains would also not reason in such a condition.
Given the right framework that can do miraculous things.
I am not saying that these systems are conscious, but they are able to abstract their context in a way allowing them to understand their own limitations.
the categories i like to use to describe what llms are capable and incapable of are: instrumental reason, which is reason as a tool for achieving a goal; and objective reason, which is reasoning about which goals are good or bad, or worth pursuing.
llms are, i think, approaching or have achieved better-than-human performance on the former category in a wide variety of applications.
the latter, not so. leaving aside that there are schools which claim (dogmatically, imho) humans don't or can't engage in objective reason, i dont believe llms are structurally capable of it. their goals can only be imposed on them from outside, coming from prompts, implicit value assumptions in training data, loss function, and rlhf. there is something about human interior experience of an objectively existing world that lets is evaluate true/false/good/bad in a way that is unique to humans among other animals.
llms can't do it. i dont just mean on ethical, epistemic, or aesthetic judgements, but even in practical circumstances like the ones engineers encounter. the reason engineers still have to work alongside llms, even though lllms are (imho) far better programmers and technicians, is that even when given a goal, there is always a graph of evaluations that lead to that objective and llms routinely fail to evaluate the tradeoffs and land in states in outcome space that are subtly (or not so subtly) wrong, even though the objective is complete!
Well, it's not doing hard reasoning like Lean or Prolog would do, but it does an approximation of that, as it was trained to reproduce linguistic patterns that encode reasoning.
Both Marky and Textise work in some places where the other doesn’t.
Obviously you (or your LLM) can throw together JavaScript bookmarklets to convert whatever page you’re on. The tools’ homepages might have premade ones, I forget.
Friendly reminder that y'all could be reading this, or you could take things a bit more seriously and check out prof. Robert Sapolsky's excellent (2024) series of lectures on human behavioural biology. (eg. on YouTube, unless your're a lucky californian)
Anyone who wants to venture an opinion about this or that reasoning should be interested in finding out about their own hardware & software first, right?
I wonder if as a hack, some of the shortcomings mentioned could be addressed through prompting.
E.g. "Approach this problem iteratively. As you form a hypothesis, track the confidence you have in various explanations you're considering, what evidence you're weighing to support each, and the unresolved questions you're holding onto. Log all that for later inspection.
Be methodical when evaluating evidence and only accept facts you have verified. At every stage, gauge how much each possible next step resolves uncertainty, and discard options unlikely to advance progress. Divide the functions I described into subagents responsible for each, and coordinate with them as you work."
If the assertion is false, this is helpful, as instructing it to reason better will cause it to reason better.
However, if the assertion is true, then no amount of prompting can solve it - you cannot explain to a fish how to use a bicycle. Telling an LLM to weigh evidence only works if an LLM can, but isn’t, weighing evidence: if it cannot do so, instructions will generate the appearance of weighing evidence with additional “thought” tokens copying that of reasoning texts, but the output will be equally groundless.
The worry I would have is that an LLMs stated "confidence" is probably not calibrated well - maybe asking it what evidence supports its conclusion and what evidence would change it would be a better approach?
They're not part of an evolved biological organism, so why would it be surprising if they're different? It is called artificial intelligence, and it's not a simulation of a brain like the Nematode OpenWorm.
> The gains have proved real, above all in mathematics and coding. But unlike AlphaGo’s search, this does not introduce a genuinely separate reasoning mechanism: The intermediate reasoning is still produced by the same next-token prediction process, iterated for longer before the model commits to an answer.
But aren't external tools used, which implement hard reasoning? Like in the case of mathematics proof work, external theorem provers?
The LLM is literally not doing "thinking"; it's just throwing shit at a theorem proving wall, until some of it sticks.
Heh, some people get really mad and downvote when you point out we don't reason very much. Forget reasoning, we have a hard enough time being rational.
>Three shortcomings prevent what chatbots do from qualifying as reasoning
>First, these models typically maintain no explicit, persistent, and inspectable epistemic state.
>Second, they lack a clean separation between what the system knows and how it manipulates that knowledge
>Third, while the chains of thought chatbots produce look like deliberation, research has demonstrated that the bots often concoct them after the fact
I think humans are guilty of all three of those? If you use that to say LLMs can't reason you may be saying humans can't reason either which is kind of unreasonable.
The author's thesis might we better put that we could increase the reliability and openness of AI by building it so we can inspect its data, assumptions and reasoning process.
‘LLMs don’t reason’ seems to equate reasoning with explicit, auditable deliberation. LLMs can draw inferences from premises, even if their explanations do not reliably reveal how they arrived at their conclusions. The article makes a case for more transparent and reliable reasoning, but does not establish the absence of reasoning altogether. The more substantive question, in my view, is whether their representations have semantic content, what could ground that content, and how this affects their reasoning capabilities and limitations.
I'm not sure I understand this. Do we have evidence humans have an independent set of beliefs not shaped by knowledge and reasoning? If so, where do these come from?
I'm especially confused about a prior statement as a scientist:
> Three shortcomings prevent what chatbots do from qualifying as reasoning (in a way that a scientist might recognize).
How does a set of beliefs help with reasoning?
> Third, while the chains of thought chatbots produce look like deliberation, research has demonstrated that the bots often concoct them after the fact, reaching an answer by one route but reporting another.
We also often do the same as humans.
reply