Right, it's an illusion of understanding. There is some sort of symbolic understanding, but that is completely due to the fact that the training data was made by humans who actually do understand, can interact with the world, and can write their thoughts down so that the LLM can insert some sort of reference to "basketball" and "Michael Jordan" in their embeddings or whatever.
I think Searle's Chinese Room argument refutes this. LLMs are simply manipulating symbols, they do not have semantic understanding. This is why hallucinations exist. And Searle's argument extends even further than LLMs.
You are basically arguing for a functional account of consciousness, but things like this have been debated for literally decades/centuries in philosophy.
Millenia, in fact. The big difference, of course, being that we now have experimental philosophy machines (aka computers). So we can actually put some of these theories to the test, and recognize how utterly inadequate most of the work done on the subject has been. We had a pretty good idea anyway, so it's not a big surprise. Theories of mind have evolved dramatically in the late 20th century. And it's pretty clear that theories of mind will have to be re-done all over again with the advent of LLMs (particularly current-generation LLMs).
The problem with the hallucination argument is (1) that is much less of a problem with good current generation AIs, and (2) living conscious breathing human beings also have a disturbing tendency to make shit up, too. So a tendency to make stuff up doesn't really serve as a disqualifier for consciousness.
Also worth mentioning that the guiding rule of what's philosophical or not is whether it's actually useful. Actually useful philosophy usually becomes something else. Usual some scientific discipline or another. And as it turns out, theories of mind are likely to become extremely useful in the near future. Expect huge advances!
1) Good current generation AIs are specifically trained to reduce hallucinations. If we had new AI system that happened to not have hallucinations as a side effect of their training, then it would be convincing. But here, it looks like we have built a pocket calculator that answer 7+13 = 14, and on top of it, we added a layer that says "if the input is 7+13, then replace the output by 20". This pocket calculator still does not know how to calculate, we just added a layer to hide its mistakes.
2) Not only "make shit up" is not the same as "hallucination" (either "making shit it" is done when the individual knows it is unreliable, or when the individual was given wrong inputs), but the point is not to say "hallucination implies no consciousness", but "large quantities of hallucinations in situations where a conscious system would be unlikely to hallucinate implies no consciousness"
First, the "13+7" is an analogy. In this analogy, "13+7" is not the real question you ask, it represents _any questions_, not just arithmetic.
But secondly, did you even noticed that in my example, the system answer CORRECTLY "13+7"? So, in my example, the thing I'm talking about and I argue does not "understand" is Claude, even if it is able to answer correctly.
My point is: the "basic LLM" part is creating a mechanism that answer without understanding (as demonstrated for example by ChatGPT failing arithmetic), and the fine-tuning or the harness is just hiding the lack of understanding by adding ad-hoc correction on the residuals. And because it is on the residuals, it looses the logical links (13+7 -> 20 is "logical", it corresponds to the math logic, it corresponds to what you get when you add 13 stones and 7 stones together. The residual is "14 -> 20", which has no meaning in itself)
The ad-hoc correction is either: 1. by training the model so it learns by heart, without understanding, that the symbols "13+7" should lead to "20", 2. or by training the model to use a pocket calculator without understanding arithmetic so it can do it itself.
You can prove that the model does not understand it very simply. Let's take the normal fine-tuned model M1. Now, let's go back to the pre-tuned version, and fine-tune it so it answer "21" to the question "13+7", and use an harness that does "sum(x, y): return x+y+1". This is model M2. M2 will fail to answer "13+7" correctly, it will say "21". And yet, M2 has been trained exactly the same way M1 was. If it is true that the additional tuning "add understanding", M2 will not be possible, it will say "error, error, do not compute, you try to train me to say that 13+7 is 21, but it does not make logical sense to me". But it does not happen: the pre-tuned model has no idea that 13+7=20 is more logical than 13+7=21, and the additional tuning is just helping him returning a more correct answer while still having no idea where this answer comes from.
It is a helpful pointer for people who might otherwise assume that a well-known argument by a famous philosopher is sound without checking too deeply. Straightforward refutations can be found on wikipedia or by thinking about it.
That just isn't true, there are no straightforward refutations of the Chinese Room that are widely accepted. Philosophers disagree about it. It's highly controversial and pretending that it's decided one way or another is not a helpful pointer for anyone.
>That just isn't true, there are no straightforward refutations of the Chinese Room that are widely accepted.
Yes there is, the systems reply is the obvious and correct answer. Philosophers that disagree are simply wrong. In the end what matters is what's true or false, not how many philosophers accept something. You can check for yourself by reading the argument, following its reasoning, and seeing that it is false; and reading the systems reply, following its reasoning, and seeing that it's true (https://plato.stanford.edu/entries/chinese-room/#SystRepl). The case is similar to those mathematical or logical proofs for the existence of god, where obviously fallacious reasoning gets a pass because it confirms deeply held beliefs.
>Most of the discussion consists of attempts to refute it. "The overwhelming majority", notes Behavioral and Brain Sciences editor Stevan Harnad,[f] "still think that the Chinese Room Argument is dead wrong".[13] The sheer volume of the literature that has grown up around it inspired Pat Hayes to comment that the field of cognitive science ought to be redefined as "the ongoing research program of showing Searle's Chinese Room Argument to be false".[14]
What you are referring to is Searle assertion that "because the Chinese room concept, I conclude that every future human-made systems will be a Chinese room and will never be 'intelligent'".
I think it is an important nuance.
You have to be careful when saying "Searle Chinese room" is dead wrong: the Chinese room concept in itself is useful and not controversial, and it is possible that current LLM are "Chinese rooms", and therefore not 'intelligent'.
We could use the "Chinese room" term to denote a system that superficially mimicks human speech, but breaks down at some point and/or uses different mechanisms such that it doesn't result in consciousness. But I don't think that was the intent of the argument and it's not how the argument is generally understood in the literature, so it would just be confusing IMO.
(And you still seem to be implicitly accepting that the basic argument is valid, which would be wrong.)
> You can check for yourself by reading the argument, following its reasoning, and seeing that it is false; and reading the systems reply, following its reasoning, and seeing that it's true
You are being tedious. I obviously have done this and I disagree with you. Saying that X is logically true and Y is logically false is not a demonstration of those baseless assertions. This is not helpful, what you're saying isn't true, and what I'm saying is backed up by the wikipedia article. The bit you quote is simply stating that most literature about the Chinese Room is an attempt to refute it, which is obvious, because the people who are convinced see no need to publish saying so. The fact that people keep publishing means that they have not yet succeeded in refuting it.
Or I can simply say this: you've made a mistake in your logic. Actually, the Chinese Room argument is correct. Since you won't explicate your logic, neither will I.
That is not what a p-zombie is. The p-zombie does not have any qualia at all. If you want to deny the existence of qualia, that's one way a few philosophers have gone (Dennett), but that seems pretty ridiculous to most people.
1. Qualia exist as something separate from functional structure (so p-zombies are conceivable)
2. Qualia don't exist at all (Dennett-style eliminativism)
But I say that there is a third position: Qualia exist, but they are the internal presentation of a sufficiently complex self-model/world-model structure. They're not an additional ingredient that could be present or absent while the functional organization stays fixed.
To return to the posthuman thought experiment, I'm not saying the posthuman has no qualia, I'm saying the red "TOXIC" warning is qualia. It has phenomenal character. The point is that any system that satisfies certain criteria and registers information must do so as some phenomenal presentation or other. The structure doesn't generate qualia as a separate byproduct; the structure operating is the experience.
A p-zombie is only conceivable if qualia are ontologically detachable, but they're not. You can't have a physicalism which stands on its own two feet and have p-zombies at the same time.
Also, it's a fundamentally silly and childish notion. "What if everything behaves exactly as if conscious -- and is functionally analogous to a conscious agent -- but secretly isn't?" is hardly different from "couldn't something be H2O without being water?," "what if the universe was created last Thursday with false memories?," or "what if only I'm real?" These are dead-end questions. Like 14-year-old-stoner philosophy: "what if your red is ackshuallly my blue?!" The so-called "hard problem" either evaporates in the light of a rigorous structural physicalism, or it's just another silly dead-end.
You have first-person knowledge of qualia. I'm not really sure how you could deny that without claiming that qualia doesn't exist. You're claiming some middle ground here that I think almost all philosophers and neuroscientists would reject (on both sides).
> "couldn't something be H2O without being water?," "what if the universe was created last Thursday with false memories?," or "what if only I'm real?" These are dead-end questions. Like 14-year-old-stoner philosophy: "what if your red is ackshuallly my blue?!"
These are all legitimate philosophical problems, Kripke definitively solved the first one in the 1970s in Naming and Necessity. You should try to be more humble about subjects which you clearly haven't read enough about. Read the Mary's room argument.
> You have first-person knowledge of qualia. I’m not sure how you could deny that...
I don't deny that. I explicitly rely on it. You must have misunderstood... My claim is not:
1) "There are no qualia"
2) "Qualia are an illusion / do not exist"
My claim is: First-person acquaintance does not license treating qualia as ontologically detachable from the physical/functional. I reject the idea that experience is a free-floating metaphysical remainder that can be subtracted while everything else stays fixed. At root it's simply a necessary form of internally presented, salience-weighted feedback.
> This middle ground would be rejected by almost all philosophers and neuroscientists
I admit that it would be rejected by dualists and epiphenomenalists, but that's hardly "almost all."
As for Mary and her room: As you know, the thought experiment is about epistemology. At most it shows that knowing all third-person facts doesn’t give you first-person acquaintance. It is of little relevance, and as a "refutation" of physicalism it's very poor.
Aren't ZKPs useless for their paranoid 'children will die if they see boobies' crap because then they'd allow for a single common token to be shared willy nilly? Not to mention that surveillance is the clear government actual goal.
This is relatable, mine is somewhat similar. It feels like a very specific version of performance anxiety that unfortunately affects the most banal social interactions. It is obviously multiplied tenfold when I'm in a situation where there are actual stakes (an interview, a first date, etc), but it still applies if I am just talking to a friend of a friend at a party that I don't know very well. The stakes feel very high to me because it's our first time talking.
It's less that I need them to like me or fear being disliked and more that I am just way too conscious of the stakes and the social interaction that's happening, which causes my brain to sort of freeze up. It feels like when I used to play tennis in high school. I'd do great at practice, then freeze up and barely remember how to hit the ball in games because the stakes on each point felt so high.
If I'm around some good friends it completely goes away. If I have hung around the person enough (even without directly talking to them), it goes away. I've also had random days where I don't feel the performance anxiety and performed really well in those situations (and coincidentally some of those days I'd meet a new group of friends or a girlfriend). It's extremely frustrating. Xanax makes the performance anxiety go away completely but slows me down cognitively so I become much less witty and interesting to talk to.
Whatever it is that gives rise to consciousness is, by definition, physics. It might not be known physics, but even if it isn't known yet, it's within the purview of physics to find out. If you're going to claim that it could be something that fundamentally can't be found out, then you're admitting to thinking in terms of magic/superstition.
You got downvoted so I gave you an upvote to compensate.
We seem to all be working with conflicting ideas. If we are strict materialists, and everything is physical, then in reality we don't have free will and this whole discussion is just the universe running on automatic.
That may indeed be true, but we are all pretending that it isn't. Some big cognitive dissidence happening here.
This bogus argument has been refuted numerous times--read Dennett's book "Freedom Evolves" for one sort of response. And whether people are "pretending" something is irrelevant (and ad hominem, and not even true). The plain fact remains that all evidence and logic supports physicalism, and even if you entertain dualistic ideas like those of David Chalmers they don't give you free will, they don't counter determinism.
what else could it be? coming from the aether? I think this one is logically a consequence if one thinks that humans are more conscious than less complex life-forms and that all life-forms are on a scale of consciousness. I don't understand any alternative, do you think there is a distinct line between conscious and unconscious life-forms? all life is as conscious as humans?
There are alternatives and I was perhaps too quick to assume everyone agreed it's an emergent property. But the only real alternatives I've encountered are (a) panpsychism: which holds that all matter is actually conscious and that asking, "what is it like to be a rock?" in the vein of Nagel is a sensical question and (b) the transmission theory of consciousness: which holds that brains are merely receivers of consciousness which emanates from other source.
The latter is not particularly parsimonious and the former I think is in some ways compelling, but I didn't mention it because if it's true then the computers AI run on are already conscious and it's a moot point.
I do think "what's it like to be a rock" is a sensible question almost regardless of the definition. I guess in the emergent view the answer is "not much". But anyhow this view (a) also allows for us to reconcile consciousness of an agent with the fact that the agent itself is somewhat an abstraction. Like one could ask, is a cell conscious & is the entirety of the human race conscious at different abstraction scales. Which I think are serious questions (as also for the stock market and for a video game AI). The explanation (b) doesn't seem to actually explain much as you state so I don't think it's even acceptable in format as a complete answer (which may not exist but still)
[2] 3.3.4 Date of Issuance. To determine the record date for ownership of Common Stock shares (whether issued as a book entry or certificate), ask: Were the Company's stock transfer books or the Warrant Agent's book entry system open when the Warrant was surrendered and the Warrant Price was paid? If yes, the record date is that same date of surrender and payment. If no, the record date is the close of business on the next day when either the books or systems are open.
How can one / should one combine the concepts of a dinosaur and monetary policy of the Ottoman Empire? What differentiates verbal reasoning from logic?
I don’t know that either of those can be solved well with formal languages or logic.
Follow up in this one… I asked an LLM to give me the funniest way to combine the concepts of a dinosaur and monetary policy of the Ottoman Empire. This was the answer.
Imagine a “Dinoflationosaurus”: a giant dinosaur who has the job of overseeing the monetary policy of the Ottoman Empire. However, this dinosaur is hopelessly behind the times, using outdated gold coins that are buried in random locations, like a prehistoric central bank.
Instead of regulating currency or adjusting interest rates, the Dinoflationosaurus spends its days stomping around, either hoarding or releasing massive piles of treasure based on whether it sees its shadow, causing huge economic fluctuations. Merchants and citizens scramble to predict where the dinosaur will dig next, turning the entire economy into a game of dinosaur-sized hide-and-seek with inflation spikes tied to the beast’s mood swings.
The Ottoman economists, dressed in traditional robes, nervously try to explain to the sultan that no one knows when the giant lizard will “stimulate the economy” by smashing a treasury vault open.
Yeah but the Cheryl's birthday problem doesn't have any ambiguity like that. It's all in very simple language, the only complexity is keeping track of states of mind, which is easy to abstract away from the language
That is exactly the point I was making in my comment above. This type of unambiguous problem is best solved using formal languages - something more like quantitative reasoning. But stuff like prolog or classical automated reasoning approaches are quite brittle. They break down quickly when you start to introduce ambiguity and noise. Statistical approaches like hidden markov models that people used in these instances were the precursor to the LLMs we have today.
But I was going down a rabbit hole there. My main point is that trying to use LLMs to solve logic puzzles - that can easily be solved in prolog - is a waste of time and a failure of the imagination. The applications that should be explored and would be most fruitful are those where there is ambiguity and contradiction.
reply