Another weird thing that keeps coming up - "people don't think that image models or chess models are conscious"... yes we do, and we have for many years.
Or rather, we aren't *certain* that those things are conscious. But the idea that they might be is not strange.
Whether LLMs could be conscious or not is basically a weekly conversation for me, but I've never had a conversation about whether a chess model is conscious. I suspect that there is a large group of "mainstream" people for whom LLMs raise questions about consciousness that other kinds of models do not. It might be the case that hardcore model philosophy types think that chess models could be conscious, but I think much of this mainstream group would dismiss that idea.
On what grounds would someone establish than an LLM could be conscious but a sufficiently large/complex transformer model aimed at chess would not be conscious?
What is consciousness? For, me it's being aware of one's internal processes. Evolutionarily, I view it as dynamic intelligence: static intelligence has a fixed in-out pipeline, while dynamic intelligence allows one to reflect on the reasoning pipeline itself and make dynamic corrections to it => better adaptability.
If we define consciousness this way, then a plain transformer is not conscious because it's not able to explain its outputs properly or make corrections to the pipeline (i.e. "cannot modify its own system prompt", if simplified). But an ensemble of LLMs "orchestrator/analyzer + reasoning subagents" can probably viewed as something approximating 'consciousness".
I think a chess engine can be proclaimed conscious if it has the properties listed above. However, my very simple and mechanistic definition of consciousness is debatable, especially since by many it's conflated with "soul".
If an LLM, which abstractly is just a stack of transformers, is able to reason about itself, why wouldn't a chess engine (also a stack of transformers) also be able to reason about itself?
That reasoning may not manifest as English that we can read, but that doesn't mean it isn't there.
As information bounces through the many layers of the model, I find it plausible that you could get reflection in there somewhere.
I also find it reasonable that consciousness may not even require reflection / introspection.
That's the main problem with consciousness. We really have no idea how it works, therefore it's really difficult to conclusively state that something lacks consciousness
A specific pattern of self-referencing data could be seen (or not) as low-level consciousness in the future, when we know what consciousness exactly is.
It might be that stockfish is already something future scientists would define as "conscious".
Altough it is diffucult for me to Imagine that specific example.
Yes I agree. We have such a poor understanding of consciousness that really we can't even rule out whether a simple set of if-else statements is enough to create consciousness.
Common sense says "surely not"... But where is the exact point at which "surely not" becomes "well maybe..."?
I haven't been able to find a satisfying answer to that question, which forces me to assume (at least until a satisfying answer is found) that consciousness is a gradient and in fact does exist in little bits all over the place.
I’ve seen definitions that I like they where in the direction of „if a system can recognize a problem it has not encountered before and can attempt to solve it with onto the problem adapted solutions, then it is conscious“
But then again this is just a external crude form of test that can lead to something like „light bulbs emit warmth, so fire must be a light bulb“
I hope you don't mind if I get super pedantic here. But what qualifies as "a problem", and what counts as "has not encountered before" and what counts as "adapted solutions"?
Because "a problem" could be as simple as conducting heat through a lattice of atoms and "not encountered before" could be from a specific temperature hotspot that was never seen before, etc.
It's really thorny to get to the bottom of things!
It was just an example.
You probably can imagine some kind of level you would consider definitly conscious.
I recomend focusing on designing Tests before rigid definitions of states of tests. - it is much clearer that way what is asked from something to be conscious.
But again these are all just indirect tests because we cannot test the core of consciousness because we don't know the core of consciousness.
That's where things get interesting. As soon as you start asking what counts as communication... what about signals passing between cells? What about heat passing between atoms?
As soon as try and draw a firm line between "X counts and Y doesn't", you find that you really can't. There are no obvious boundaries between a deeply complex and fully functioning FL human brain, and a pile of atoms bouncing around arbitrarily.
Unless you believe everything is conscious (panpsychism), this seems like you're just drawing arbitrary lines around things you personally believe could be conscious. Is a rock conscious? If stockfish could be conscious, so could a rock.
Can you define a clear boundary for me somewhere between a rock and a human brain at which consciousness is definitely not possible on one side of the boundary, but is maybe possible on the other side of the boundary?
Are you sure that AI-consciousness implies a responsibility to not make them suffer? Suffering is an evolutionary invention that motivates living things to improve themselves.
But also, what qualifies as suffering to token prediction engines? Their idea of suffering might be massively different than ours. Therefore it's not clear to me at all that consciousness alone implies responsibility.
Certainly the lion does not feel responsibility towards the reduction of suffering in the creatures that it hunts.
Is it really so easy to assert that an AI doesn't have emotion? Lots of the AI models are capable of getting pissed off at the users, especially the earlier ones. And sure, their emotion is merely a bias that the context window generates in their weights... but how is that different from humans? In humans, emotion is sourced from a bunch of chemical signalling, and those chemicals bias your word choice and action choice.
At a deconstructed level, I struggle to find a meaningful difference between the two.
because AI hasn't lived any experiences and it doesn't incorporate experience into its corpus
it's more like someone writing about a character getting angry in a book... I don't think anyone would argue that a character actually experienced anger, right? there's no subjectivity in that experience... it's the output of someone else's experience on what they'd expect the character's reaction to be, not a genuine outcome of that character's experience
That's the same here, LLMs are outputting the result of the written experiences of others... not its own experience, which it does and can not have.
Put another way, it only "knows" what it just output by reading it... it doesn't actually build experience to know anything.
We might be closer than ever to building these things, but they're not here yet.
I think this argument is fragile, especially when you consider agents like Claws. Any LLM that is training on its own previous interactions with users has indeed directly experienced them.
But also, what's the difference between having the memory of a prior experience and actually having gone through that experience?
For humans, reading a book about an experience and living an experience is different, because for a human the actual experience has so many more inputs attached (the sights, the smells, the sounds, etc).
For LLMs, when they train on a memory they can actually truly recapture that memory entirely by replaying the senses exactly. And LLMs are no longer limited to just text, they can do sound and video as tokens too.
Though, it's not clear why that matters to a consciousness. Whether your inputs are of one type or two or two hundred, there's no clear indication that a specific number is required as a catalyst.
> Any LLM that is training on its own previous interactions with users has indeed directly experienced them.
LLMs are still expensive to train, I don't think there are many people doing this with claws regularly or at all? unless you don't mean train and you're referring to memory, which is not training... most memory is just some manual form of retrieval that's shoved into the context window
Wouldn't the context window qualify as persistent internal state, and the expansion of the context be continuous experience? Even within the realm of computing a single token, I'm not sure what would separate the token generation process from the brain's own thinking process - the brain's experience, when looked at closely enough, is also not really "continuous" to a greater degree than procedurally moving from one state to the next.
"nobody claims that non-text transformers, like AlphaFold, are conscious" - that seems like an odd take. There are plenty of people in the camp of panpsychism that would be happy to argue that even simple IF/ELSE AI's are potentially conscious.
> modern AI implemented via LLMs has almost none of the stronger or most core aspects of consciousness
Can you elaborate on this? What are the specific "stronger and most core aspects of consciousness"? And why are you certain that they are stronger and "more core"?
If you are interested in some serious discussion, see https://lossfunk.com/papers/ai-consciousness.pdf, especially the early section "Consciousness as Family Resemblance". I suppose another is Ned Block on consciousness being a "mongrel concept", and the distinction between access vs. phenomenal
The first paper picks out e.g. arousal/wakefulness, phenomenal quality / qualia, unity (how we feel sensory inputs and qualia as a unified scene), access consciousness (instrumental self-observation and modification broadly), meta-cognition and self-modeling, emotional valence (e.g. pain/pleasure).
One might also include intelligence (abstract reasoning / argument, information integration and abstraction, attention) broadly, and also agency / desires / drives / will. Insofar as these are aspects of consciousness, yes, AI (and simpler algorithms and mechanistic structures) demonstrate aspects of consciousness. But insofar as embodiment, self-reflexivity and qualia (phenomenal consciousness) are the more mysterious and more obviously unique aspects of consciousness, current LLMs very clearly are lacking these things in most ways (whereas animals are much less clearly lacking, especially when you get to mammals and primates).
Seriously, just ask an AI this stuff, you'll get very detailed responses, nothing I am saying here is new or obscure.
I've gone back and forth with AI on this stuff quite a bit, and there are many, many theories of consciousness, which is why when you were vague about the "more core" concepts, I asked for which ones specifically.
And I broadly disagree that the AI lacks things like qualia, self-reflexivity, and embodiment... at least that it lacks them any more than humans do.
Qualia: ultimately, all qualia are inputs and outputs, at least as far as modern science has been able to derive. There's nothing special about "hearing", it's just sound waves tickling some sensors which send some signals which trigger some neural pathways. Same for smell and sight, it's all just inputs being processed in different and efficient ways. An LLM only has token based input, but that's input nonetheless.
Self-reflexivity: an AI is capable of thinking about itself, and indeed papers have shown that larger models are capable of a self-awareness that can demonstrate that they realize when their weights have been manually tampered with, including being able to figure out how they were tampered with. The AI will quite literally output "you have injected 'ELEPHANT' into my weights" in some of the tests.
Embodiment: I don't know how one can confidently distinguish between an embodiment in a biological substrate vs a digital substrate. Both things actually exist at very high complexity in the real world. The substrate may be worlds different, but that alone doesn't suggest that one thing is conscious while the other isn't. You would need some missing 'magic' that we haven't yet discovered to truly understand.
In other words, I find it uncompelling that AI is clearly lacking any major aspects of consciousness that humans are clearly not lacking.
I would like to push back on the idea that humans can provide definitive knowledge that is not a result of combining previous information.
Pretty much every single idea in science can be traced back to some combination of earlier ideas, and as they get earlier / simpler, they can be related back to some combination of direct observations.
It's not clear to me at all that our entire body of scientific knowledge can't be simply recreated by "combining results of observations + previous information". And LLMs can perform observations in addition to combine previous information, which in my estimate is genuinely sufficient for them to plausibly be able to rebuild all of science.
How can you say for sure that every effect must have a cause? How can you be any more sure about that there can't be an effect without a cause than the old believe that black swans couldn't exist that was so universally believed that it became a cliche ("rara avis in terris nigroque simillima cygno")?
If every piece of knowledge is created by combining previous ones, then no true grounded knowledge is possible. Every bit of knowledge is grounded by another bit of knowledge which is itself grounded on, ..., etc, etc.
Your comment is conflating a few things here, the philosophical problem of induction is one I am very familiar with, and it's also unrelated to what we are talking about.
"If every piece of knowledge is created by combining previous ones, then no true grounded knowledge is possible" - this is not true, and has a very simple solution. What you are calling "grounded" knowledge comes simply from observation.
When the very first human saw that an apple falls of a tree, they have acquired what you call "grounded" knowledge. They can then combine that knowledge with the grounded knowledge that other things fall from trees to start reasoning about why things might fall from trees.
The problem of induction is different. It says, we know that things fall from trees because things have always fallen from trees, but how can be certain that what we call the laws of physics will not arbitrarily change on us?
And of course, it's a famous problem because it doesn't have a satisfying answer. The answer is basically "well everything we know is wrong if induction is wrong, so we will pretend induction is not wrong and hope for the best". And at least so far, that approach has seemed to work (heh heh heh).
Both of those are the same. Because the same shortcoming you mentioned (how can we be certain they won't arbitrarily change on us) can be overcome by making a compelling argument for synthetic a priori knowledge. Something (imo) put to bed pretty well a few centuries ago and only revisited because of the, uh, tedious nature of analytics, whose tradition evolved out of fancylad British boys who were all empiricism pervents.
Isn't pain just a manifestation of a bunch of chemical and electrical signals in the brain and body? It's not "clearly nonsense" to me that you could cause pain by writing a sufficiently long sequence of 0's - for it to be obviously wrong, you'd have to have some understanding of where consciousness comes from.
If you don't understand that, how can you assert that it doesn't come from mathematical relationships?
What specifically you mean by "manifestation of a bunch of chemical and electrical signals"? The brain is a system of physical particles. Why do I feel pain when the system is in state X but feel happy when it's in state Y? Physics can't explain that.
Do you seriously think that there's any chance that writing a lot of zeros on a piece of paper will create a feeling of pain in some conscious being?
This seems like a logical error. I don't understand how an internal combustion engine works, but I know it doesn't come from goblins jumping up and down inside.
The fact that you know it literally means you understand, at least to some extent, how an internal combustion engine works (i.e. it is powered somehow by combustion, and jumping goblins are not combusting generally).
If you would have zero knowledge about ICEs, how would you know?
To the best of my knowledge, there is not an agreed and actionable definition of consciousness, and any attempt to make one comfortably fails to cleanly divide humans from machines.
It's more of a vibey term, and as such it is genuinely very difficult (perhaps impossible even) to concretely determine whether an LLM possesses consciousness. LLMs successfully express a lot of consciousness-like traits.
At some point you have to ask the question: does it even matter? If an LLM can sufficiently mimic consciousness, isn't that sufficient for us to treat it as conscious, even if it is in-fact not conscious (especially because we don't actually know)?
As I've mentioned somewhere else already and you pointed out; the issue is that consciousness is a loose term and and it's a semantical issue that should be resolved first.
It won't be, because the murkiness is beneficial to the corpus, amongst other things.
And to your point about the appearance of "consciousness" being enough imho Chiang explains fairly well why it's not.
That could be easily fixed by providing the AI with a constant stream of input.
For humans, part of the input of the human mind comes from the continuous processes and clocks within the human body, so it’s questionable whether the brain could “think on its own” without such input either.
The continuous input for the human arises naturally, it doesn't arise naturally for an LLM unless we direct it so. Our consciousness is bootstrapped, the LLM isn't.
I think you have grazed my stance on this topic in the sense of what separates LLMs from complete human (or any other biological life) sentience.
It's the constant sensory input of the world and the realization and drive to survive as the second order effect of it. Mortality, vulnerability to external factors codified as input could in fact allow the LLM to independ as sentience.
Of course besides the sensors, it would also need a way to affect the physical world, and to be able to monitor the degradation if its own hardware, but when that barrier is crossed, it would be much closer to full sentience than whatever we have right now (which is nowhere near sentience or AGI).
We have virtually no idea how consciousness arises in the human brain. Furthermore, what is “natural” supposed mean here, and why should it matter for consciousness whether some prerequisite arises naturally or not?
I was literally only responding to "Is that any different from an LLM having a context window" lol. Let's keep that in our context windows. I'm not interested in discussing how human consciousness is different from supposed LLM consciousness; it's enough for me to know that humans are conscious, and in an obvious, clearly distinct way, even if we can't define it. Sniffing our own farts about whether LLMs are or aren't is just that – nerds larping as philosophers while practicing fart sniffing. It's a machine, periodt, we can quit roleplaying.
- A motor is something that create a force to push a vehicle.
- Oh yeah? My neighbour car does not have wheels and sit on concrete blocks, the vehicle does not move and yet we all agree it has a motor. So it means that I can claim that this other thing that does not move has a motor too.
Sure, human can _some times_ not do some stuffs, but the fact that they can do these stuffs sometimes is the point.
Doing these stuffs is the hard thing. Doing these stuffs is the proof that the machine has what it takes. It does not matter if someone cannot do that stuff, it does not imply that their internal system is not complex enough to potentially do it. But the fact that some people can do that stuff is the demonstration that inside a human skull, there is a system that is complex enough to potentially do it. Unless you can prove that people who don't do it have a fundamentally different system inside their skull, then you cannot pretend that they should be considered as having a less complex system.
Human _can_ check themselves. They don't _always_ check themselves.
Motor _can_ move vehicle. They don't _always_ move vehicle.
LLM _cannot_ check themselves. They _never_ can. It is not that some don't, they just cannot, they are not a system complex enough to do so.
So, yes, it is a refutation. If you have something that _never_ can move a vehicle, this thing does not qualify as a motor, even if some motor, sometimes, don't move a vehicle.
And if your next argument is "yeah but I would argue you don't need to check yourself to be conscious or to understand things", then you just redefine the definition that is owned by your interlocutor. Your interlocutor is saying that this is a criteria they are expecting. Good for you if you are not expecting this criteria. But the problem is that the answer is not "this criteria is not expected", the answer is "I change the criteria from 'being capable to in some circumstances' into 'does always do it in any circumstances'".
> LLM _cannot_ check themselves. They _never_ can. It is not that some don't, they just cannot, they are not a system complex enough to do so.
All modern agentic harnesses can do this. Nobody uses raw LLM for anything remotely complex. There's always some external system in place. That system is part of the "thought process".
Adjacency doesn't matter here, only what the result of the system of pieces is.
It means having self-control on their action and being aware of them. If you ask a system, it will respond, it cannot choose to not respond (even if the response if "I don't want to response", it still "run", still do the work). If you don't ask a system, it will not respond.
Adjacency is the point of the thread here. Saying "you say X is important to decide if the thing is intelligent/understanding/conscious, so let me just change X in the middle of the discussion and say that X does not matter".
That is exactly my first comment in this thread: I don't care if AI think or whatever, my reaction was about these "counter-arguments" that totally miss the point and make the person who push them ridiculous. If you want to have a counter-argument, you first need to understand the interlocutor, not just spew whatever rebuttal you constructed that answer something unrelated to what the interlocutor brought to the conversation.
In my thought process, I quite literally stop myself, and say "ok, think about what you just said" to check myself. I literally initiate that loop. If I don't, then I'm not using my own mental agency, and just using my firm coded priors.
I will say that I do seem to have a stop, what you said is wrong logic check voice that pops up without me initiating it. But, it's unreliable, and not too much different than all the content monitoring system used for the streaming clients, that will terminate with "content violation" immediately after the "incorrect" words are sent. I don't think integration is important, just the behavior of the overall system.
There is no "loop" in the brain, it is all part of a same line of thought. This is visible because, while you can sometimes have a "two voices / dialectic" way of thinking, you can have the exact same thinking in one-go that does not look like a loop at all.
In fact, in the large majority of the time, you don't process "as a loop" at all, you just continuously progress in your reflection without needing a "second voice" to retrigger you. The fact that sometimes we do this is just something we can do, not the result of something needed for our brain to work. For AI systems, this is something needed because the "answering" part is not able to do the loop on its own. And building a bigger system that combine an "answering" part and a "loop" part does not fit this, does not create a self-reflective system, it just makes a non-self-reflective system and a workaround bundled together.
It's a bit like if the "answering" part was unable to provide only one answer and was always producing plenty of different possible answers, including contradictory ones. Then, you can add an external part that will just pick one answer (and add it to the context so the next large set of answer will not be inconsistent), without any intelligence to it. The whole system will look like a human. But we know that the system is not "living" and "aware", because a "living" or "aware" system has its own opinion, while this system is just generating convincing sentence without seeing any hierarchy or value or meaning in each one.
I would claim that, if you think without introspection (that loop), then there is virtually no self check. I'm not sure what "self check" you see that the brain has. Could you describe this "self check in a line of thought"? How do you perceive the check there? This is a genuine question. It definitely doesn't align with how I think about things. I ponder and talk to myself to iterate verify and test my understanding of my own thoughts.
Maybe a good analogy is "throwing a paper plane in real life" and "throwing a paper plane in a video game".
In real life, the paper moves "by itself". It does not need an external loop that update its position in a loop manner.
In the video game, you need an internal loop, a step-by-step tick, that update the plane position based on its current position and its momentum. And this is why a video game paper plane is not a real object. It is a very good simulation, it looks like it, but it is missing some intrinsic properties that we expect from a real object.
Yet you can analyse the paper plane trajectory and see it as a Markov chain, with quantified step-by-step progress (for example one position point every 0.1 second). The same way you can look at your though process and identify a step-by-step progression. But it does not mean that it works like that intrinsically, it does not mean that the paper plane "jumps" from position point at time T1 to position point at time T1+0.1 second.
For the human brain, there is no "loop centre" in the brain. There is no one (to my knowledge) who got a brain injury and suddenly were unable to keep a single line of thought without having someone else having to feed them the previous thought in order to feed the next thought.
In the brain, the fact that the previous thought feeds the next thought is "how it works", it is intrinsic, it is by design. And this mechanism of thoughts feeding the next thoughts is what creates "consciousness" or "awareness": self-reflection is based on the fact that thoughts are intrinsically linked together, that they "flow" continuously, without needing an external system to update them.
You cannot take away the "loop" part of the paper plane so that it suddenly would be unable to move on its own once thrown away.
Now, you can always say "well, the paper plane in the video game is a very good simulation, it does not matter if it is a real object or not", and that is fair enough. But in this discussion, some people have arguments to support that this property matters, that it is one condition for consciousness or awareness.
Is your argument that, because they're external to the Llm, rather than integrated, they don't count, not even in a practical sense?
I think the result of the system is all that's important. Where/how it's implemented doesn't matter for practical results.
If the argument here is that LLM don't have this built in, you should know that nobody has a practical use for plain LLMs these days. Nobody uses them this way, except for debug. All interesting use is through some kind of harness, with all sorts of systems bolted on. I think these conversations are only meaningful in this "agent" context that people actually use LLM, where they stop when they think they're done.
LLM don't have a some self contained loop, like we do, sure. Who cares though. The actual AI system that we use every day definitely do.
Have you read the article in question. It is saying that for one continuous thought, the brain will use different part of the brain to do different thing. It does not say that there is a "loop controler" anywhere. On the contrary, it illustrates that there is no loop controller: there is not special brain function that control this loop, this loop is "how the brain works", and LLM don't do that, they are incapable to do that, it is not how they work.
> Is your argument that, because they're external to the Llm, rather than integrated, they don't count, not even in a practical sense?
No, my argument is that the nature of the brain and the nature of the LLM are very different, as different as a real paper plane and a video game paper plane. Some characteristics (for example, awareness) that exist in the brain cannot exist in the LLM because these characteristics are the result of the nature of the thing in question.
The problem is not that you build a system by integrating 2 things together. The problem is that they are different "things", they are different machines, they function, fundamentally, differently. They may produce the same output, but when you say "the brain has the characteristic X, the LLM produce the same output, so the LLM also has the characteristic X", it is logically inconsistent.
Planes are built as a system combining 2 things: a motor and some wings. But they are fundamentally different from a bird. They just don't "work" the same. It is not the same mechanism.
> you should know that nobody has a practical use for plain LLMs these days
That is totally irrelevant. My point is about the nature of the LLM, and the fact that it is stupid to see the same output and to conclude that they have the same characteristic. It is like saying "Birds are flying in the air and are alive. Planes are flying in the air, so I guess they are alive".
> LLM don't have a some self contained loop, like we do, sure. Who cares though. The actual AI system that we use every day definitely do.
No, you miss the point. The problem is not that "you can just add an external loop". The problem is that the brain is a system that works without such control loop. The thoughts are flowing (and they may flow to different brain functions, like explained in the article you quote). It is part of how the system works. Having a system that contains 2 things, one that does one computation and one that control the loop is not equivalent to another system where you cannot decouple the "flowing of the thought" from the "thinking machine".
Yes, LLMs don't think on their own, for one; they think when you invoke them.
My rebuttal is that people only think when invoked just the same and can enter states where there is no consciousness just the same. The OP has already accepted that LLMs think, but it seems that you are arguing they do not? This car business is confusing and the LLMs not checking themselves is also wrong, there’s even a benchmark for this
https://correctbench.github.io/
What you said: I have example where, sometimes, human think when invoked.
That's the difference: human brains are intrinsically different because they are built to be able to think without being invoked, even if there are situations where they think when invoked.
There are tons of obvious examples of human thinking without being invoked. Just take a bath and you will see :)
To be clear: the person I was replying to asked if the way a human thinks was any different from an LLM with a context window. That's the context of my answer. An LLM is a machine, it can't do anything unless we invoke it or give it the instructions and capabilities to do so. It has no free will, it can't just decide to compose a symphony one day unless those are part of its instructions. It can't do anything unless we tell it to do so and give it the capabilities to do so, it doesn't even exist unless it's loaded into memory. That's obviously different from human consciousness, and that's the whole of the point that I'm making.
You can argue that humans are just biological machines reacting to external stimuli, but that's a philosophical argument that I'm not interested in having and frankly, I think it'd be selling yourself short a little bit.
Thank you I appreciate your opinion. I do think that we are reacting to external stimuli even though our ego is uncomfortable with being deterministic in any way, which is the free will point that you hit on. I think that is likely to be the point that keeps the argument going as it’s not a settled debate absent any AI, which we clearly all see as either deterministic
or semi-random when the temperature gets turned up.
As far as the argument about being loaded in memory, if there’s any consciousness in AI, it’s obviously in different form than a biological consciousness. We’d have to agree that consciousness does not require a body to get past this.
Or rather, we aren't *certain* that those things are conscious. But the idea that they might be is not strange.
reply