I think the moat that China has is energy costs. It's taking learnings from the Bitter Lesson. If you role up scale and compute to the next level, it's energy resources. China has it and sharing open weight models is an effective means of removing the tech moat. This idea has been floating around for a bit now (I'm not taking credit for it).
It's not energy costs. The US produces about 70% more electricity per capita. Chinese households do pay less than half what US households pay for electricity, but that's because the NDRC sets prices below costs for households. They make it up by charging industry more, and the industrial electricity prices in China are roughly 34% higher than in the US.
per capita seems the wrong metric given the difference in population sizes and America's wealth. They have roughly 4x the people and have added 10x new power capacity in the last 10 years, not to mention lapping us in renewable and long distance transmission lines added.
I can’t help but notice how much this echoes Francois Chollet’s On the Measure of Intelligence: https://arxiv.org/abs/1911.01547
Most of frontier-model progress still looks like skill acquisition optimization: broader benchmark coverage and performance, more domains absorbed into the training distribution, and increasingly strong performance within that surface area.
It seems more about coverage-driven competence. Somewhat analogous to overfitting at scale.
The harder question, in Chollet’s framing, is: how efficiently can a system learn to do something genuinely new?
With our current AI architectures and training in place, I think we will only continue on skill acquisition optimization vs. truly novel intelligence.
1) First, you are talking about positive forward transfer in continual learning. I've been giving talks for the past 6-7 years about how that community (I was one of the founders) went astray and wasn't focusing enough on that topic, but continual learning of the kind you are thinking isn't in any of these systems right now. I think some people left the Grok team to make a start-up to focus on that. By forward transfer, what I mean is weights update over time and past learning improves future learning such that we get better sample efficiency.
2) Psychologists distinguish among different kinds of intelligence for Spearman's g (IQ). Crystalized intelligence is using already acquired knowledge (frontier models probably have maxed out that). Fluid intelligence is reasoning and finding solutions in novel situations or without the necessary crystalized knowledge. [Giving colloquial definitions]
3) Now, interestingly, neither of those are correlated with _creativity_ (just they are independent, note some have this threshold theory but it hasn't held up in recent papers). That's what the AI's really are terrible at -- creativity. But I'd argue the vast majority of humans aren't very creative, with truly out-of-the-box ideas. Given that this is HN, and a non-trivial number of us have ADHD, creativity is positively correlated with ADHD.
I did a bunch of research on these topics for my AGI course that I teach each Spring (where I then point out conflicting definitions and start using multiple alternative terms rather than AGI to distinguish among the different definitions).
It should obviously not come as a surprise that having areas of the brains working differently can explain an awfull lot of things... but to answer your question directly, yes of course of course there are!
Simply look up all the many, well
-supported and -researched known correlations with ADHD first. In the second step, you can construct the set of all possible correlations, and subtract the well-researched ones if it. What is left is the set of correlations that are either not explained by ADHD (the big majority I would assume) or explained by ADHD, but as-of-yet unknowingly so.
This might be a bit anticlimatic, but clinical psychology is pretty straightforward study design and statistics, and set theory is not that new either, so... no big surprises I am afraid.
Not sure the thrust of your comment, but what is really going on is just novelty seeking to get a bit of a dopamine rush, which slides into creativity.
It all comes down to dopamine at the end of the day.
> That's what the AI's really are terrible at -- creativity. But I'd argue the vast majority of humans aren't very creative, with truly out-of-the-box ideas. Given that this is HN, and a non-trivial number of us have ADHD, creativity is positively correlated with ADHD.
I think it's hard to define creativity in the context of AI because they seemingly just make up new hyphenated terms for everything. Is that creativity? If not, what about when they do the same thing different ideas in the latent space?
If we say that simply nailing one concept to another isn't creativity, then AIs are incapable of creativity, while the vast majority of humans are incapable of creativity. This is just a long way of saying "0 AIs have creativity, 0.00001% of humans have creativity", and the difference between zero and a very small number is infinity.
It’s not that they’re incapable of creativity, it’s that LLM-driven creativity is terrible, and nothing makes me cringe more than when it uses a word in a “novel” way.
But monkeys-with-typewriters, they sometimes stumble upon something that doesn’t suck. But if you don’t want to spend a fortune retrying the same task until you get a suitable result you have to inject your own taste.
Sometimes I start with a super vague prompt and see how close agents can get to something that doesn’t suck. I inevitably get frustrated about 6-7 prompts in when they’ve created a complete mess because they have no taste. So I restart and inject my taste into the process. Things like linters, test suites, which 3PLs to use, etc.
Taste or vision? They are hill climbing and they can't see the other side. And sometimes that is because they don't live in your head and don't know what you want.
Very insightful actually. Taste and vision are both instances of holding a model that predicts a good result beyond the threshold of validity in theory but reality happens to align. Is it luck then? Perhaps meta-luck where lucky weights produce “accidentally great” results with some predictability.
Steve Jobs had a mental model that brought the iPhone. No one really wanted it but something in his life biased the result.
So creativity is having weights so good you can project way out into latent space beyond what is reasonable.
I hope you’re not actually trying to compare human cognitive processes to an LLM, because that would be incredibly reductive, not to mention lacking in any empirical grounding.
Nice to get some resonance. And if I may, past --> taste; vision --> future. And to compensate for our poor memory, taste<=>value network while vision<=>policy network, borrowing from RL parlance.
> That's what the AI's really are terrible at -- creativity
Good thought piece here "We Are Losing the Ability to Discover What We Didn’t Know to Ask[1]" By Anne-Laure Le Cunff
It keeps playing on my mind as I see people at work follow some predetermined AI workflow to get their jobs done, the art of being curious and exploring around the problem is so important to the really big innovations. Been thinking about how to address this through some of the harnesses we are developing in the knowledge working space.
I met a senior (as in, 4th year of college) recently and she asked me: what advice do you have for someone just graduating in the AI age? (as I had told her that I've been in ML for 20+ years, etc.)
My recommendation to her was: just _play_ with the AI! It's a brand new tool, and none of us knows its capabilities, limitations, boundaries etc. (which are fluid, of course). So just spend as much time as you can tinkering with it, playing with it, making it do things it was not expected to do, etc. and you'll develop an idea of how to make better use of it.
Tangent: Anne-Laure Le Cunff is a neuroscientist with a talent for community-building and writing, and bringing evidence-based approaches to bear on practical solutions to various domains. See eg https://nesslabs.com
Hi thanks for the insight. Do you see a role for Control Systems(i.e. ones analogus to Instrumentation engineering) playing a role to modulate certain parts of continual learning? One very important way we learn are lived experiences, it's like telling memory:this part is more important( for emotional or social utility values), pay attention. Good or bad lived experiences both count. I guess is that a path that practical research is considering?
Appreciate the input! Responses below to your first two items:
1) I'm leaning more into a broader sense, which is that given the priors a system already possesses, how efficient can it acquire competence on a novel task? If I'm reading the point you make, you're focussing on continual learning right? If so, I'm not necessarily restricting my statement above to that.
Here's another reframing: How much of the benchmark improvements come from overwhelmingly large training distributions vs improving the models for adapting to things genuinely outside of it?
2) Excellent points about crystalized and fluid intelligence. Wouldn't the LLM scaling gains be a representation of crystallized capabilities? In regard to Gf, that is exactly what I am asking about. That is what seems to be lacking, Gf like adaptation under genuine novelty.
My concern is that it's increasingly difficult to tell of what looks like Gf like behavior is really coming from better adaption vs. having broad priors from the model's large learned distributions.
> weights update over time and past learning improves future learning such that we get better sample efficiency
Is there an architecture-independent definition of forward transfer?
For the practical experience and implications of AI progress, I think we are increasingly discussing what these LLMs can accomplish inside a stateful harness, the state of which could be described as part of a (very squirrely) parameter space.
"Given that this is HN, and a non-trivial number of us have ADHD, creativity is positively correlated with ADHD." Is this a statement of fact? As someone with ADHD and pretty confident in my creativity, I still believe this is much more cope than fact(which could be more a self-doubt thing than anything else).
Further, if there is a correlation, I'd bet it's not so much an intrinsic "creativity" trait, but more effectively higher creativity because more trials. That is, along the lines of Chollet's paper, a measure of creativity should be based on a fixed budget with fixed knowledge.
Among many other possibilities I haven't considered, perhaps another mechanism could be that because ADHD people spend more time thinking in less goal-oriented ways and mixing thoughts on accident, perhaps we do in fact gain some learned creativity via experience with vagueness[1]? But that might also imply that part of creativity is actually being able to diffuse more freely through thought space and lowering the barrier to attempted connections between ideas. That lower barrier leads to less likelihood of any "collision" being meaningful but maybe it's overcome by higher collision rates? Or maybe effectively higher order (not just pairwise) collisions?
Disclaimer in case it's not obvious: I don't know any of the literature on what creativity even means or how it's quantified.
[1] Which is me injecting an assumption that creativity ~= connecting things with no obvious or well-troden reasoning path between them.
edit -- oops just looked at your profile after seeing someone elses comment. I assume you are stating a fact then, leaving original anyway
I think it's just that many of us don't have the capability to just do things in rote or the conventional ways, instead we must rely more on the creative and unconventional ways of doing things and parts of the brain responsible for that
This comment and the one above from astrobiased feel like coming into a messy codebase, and it’s more work to sort it out than it would have been to write it from scratch.... And since I actually do intelligence testing as a clinical psychologist, I have experience with this in both practice and theory. So now I’m going to waste an hour because I just have to respond to “something is wrong on the internet.”.
Chollet's distinction is useful. High performance on known tasks is not the same thing as efficient adaptation to a novel task. Prior knowledge and training data can buy skill. That is a central point of On the Measure of Intelligence. But it does not follow that current frontier progress is only "coverage-driven competence." That is a hypothesis. It is not a result established by Chollet's framework.
"Overfitting at scale" is also the wrong term. A model that learns broad representations and applies them successfully to unseen examples is generalizing. The relevant concern is whether apparent novelty is actually inside the effective training distribution, not whether the model is "overfit."
There is also an unstated premise here: that adding broad knowledge and skills cannot improve the machinery used for novel problem solving. I do not see a basis for assuming that. Learned representations, abstractions, reasoning patterns, and cross-domain analogies can themselves support transfer to new tasks. Whether this becomes sufficient for general intelligence is an open question with insufficient data. But its a perfectly valid hypothesis right now that, given enough domain knowledge and symbolic reasoning examples, LLM COULD maybe "Grok" AGI at a certain critical threshold.
And ARC-AGI-3 was specifically designed around novel abstract environments that require exploration and adaptation. Astra scores 99.9% with OpenAI's context-preserving Provider Adapter, and ARC reports that Astra constructed compact symbolic models of unfamiliar environments. That does not prove AGI, but it points in that direction more so than the other way around.
Gc roughly maps to acquired knowledge. Gf roughly maps to reasoning in relatively novel situations. Naming those two categories does not tell us whether increasing acquired knowledge and learned abstractions in an AI can improve Gf-like behavior. That causal question is exactly what is disputed.
And "Frontier models probably have maxed out crystallized intelligence" is just obviously wrong, unless you think they have been able to dig up every a scrap of paper with knowledge/information on it in the entire world, AND that there is no more useful knowledge to be generated left in the universe.
And the statement that intelligence and creativity are independent is simply wrong. A meta-analysis of 112 studies and 34k participants found a positive correlation of about r .25 between intelligence and divergent thinking. It also found that using g, Gf, or Gc did not eliminate that relationship. Creative achievement has a smaller but still positive meta-analytic association with intelligence, around r = .16. These are distinct constructs, not independent constructs.
And this is just a bad take: "AIs are terrible at creativity". At best that depends on which creativity, and I think its straight up wrong. On divergent thinking tasks, the operationalization behind every ADHD study you could cite, LLMs score above most humans, with the top humans still ahead. If you means Big-C, paradigm-shifting creativity, that is a different construct and none of the ADHD evidence transfers to it.
And if I where to say what I subjectively feel and see.... I have ABSOLUTELY no idea how people can say that we are not seeing sparks of creativity from AIs already. If a PERSON produced some of the music, solutions or deductions that I have seen AIs do, people would have NO problem celebrating it as extremely creative.
And finally, the ADHD claim is also, at best, overstated and just as often debunked. There is some evidence that higher subclinical ADHD trait scores, often survey studies only, are associated with better performance on some divergent-thinking measures. But a review of 31 studies did not find a consistent creativity advantage for people with clinical ADHD, and it found no evidence of better convergent thinking.
Okay, I’m done… And nobody noticed that I’m not doing my job here.
For me a basic test of A.I. creativity is give the A.I. chapter 1 of a novel it has not seen and ask it to write chapter 2, then compare the quality to the original author's chapter 2. The A.I. is always terrible at this in terms of matching the author's level of quality.
I would think that’s the opposite of creativity. Rather, I would label that as pattern matching and prediction capability.
Why box in creativity basically as the ability to mimic someone else who is creative? Why not choose something that better matches the definition of creativity (the ability to make new things, think of original ideas, or show imagination that is novel, useful, or pleasing)?
And even then, I am assuming the premise that the original novels are good and creative, especially chapter 2. Most novels are not very creative.
If it could make 5 different versions of chapter 2, all with different directions for the story, and all with novel and interesting developments, wouldn’t that be a better definition of creativity than “can it read chapter 1 and be able to copy style and deduce/predict what the author is going to do in chapter 2”?
And also, that’s only a subset of creativity (storytelling). I’ve met plenty of people who are terrible at writing and storytelling but can come up with the most impressive and novel solutions to a practical problem instantly.
Final thought: I have been following the development in the anime AI generation scene for a while now. It’s not even close to anything anyone would call creative or even OK quality. But it’s also massively impressive that it’s moving in that direction really fast. And if you watch enough, you are going to start to see some truly creative sparks. And some of the mainstream stuff that gets created and labeled as creative… really... How many isekai series with the same story have humans not made already?
> If it could make 5 different versions of chapter 2, all with different directions for the story, and all with novel and interesting developments, wouldn’t that be a better definition of creativity than “can it read chapter 1 and be able to copy style and deduce/predict what the author is going to do in chapter 2”?
If I like the novel, an alternative version where things happen differently would be fine.
The problem isn't that it's different. The problem is quality.
A lot of Isekai stories are crap but you can still rank them in terms of the author's ability or inability to have a creative POV that elevates the material.
I think “what I individually like” is a poor measurement of creativity, at least alone.
And yes you can rank quality. My point was that if you made two bell curves of the distribution quality and creativity of all new manga, The one for “AI slop manga” allready started overlapping with “normal human manga”.
That’s just a fancy way of saying the absolute best AI slop is at the level of the absolute worst human creation.
The interesting thing is that the AI slop curve clearly is moving to the right every month. Where it will stop tho, impossible to say.
Is chapter 1 of a novel really enough context to generate a chapter 2 of sufficient quality to match up to an author who spent a good amount of time planning out an entire story and whose manuscript probably went through a lot of revisions?
I don't necessarily just feed it chapter 1. (And chapter length varies between novels).
And no I don't think asking the A.I. to write the next 2000 words would require it to have the entire novel planned out. Not all writers even outline in advance.
My take on it is that even if there was no "novel" discovery (leaving that up to the reader to define), if you consider human knowledge to be a sphere in N-dimensional state space, "within that surface area" is Swiss cheese, and AI seems to at minimum be able to fill in some of those holes.
And those hole-fillings, for all intents and purposes, look to us like novelty, even if much of it was simply overlooked by us, or, perhaps, unable to attain due to time or other constraints.
Now if you want to talk beyond the sphere, let's call it the "novel novel discovery of the unknown unknowns", then you may have a point, and AI may be more limited than humans in discovering the things that we don't know we don't know. Especially the as-yet-unmodelable things i.e. intuition.
Plenty of discovery left just working from first principles, however. Which I cautiously suggest current frontier AI is good enough to model to a significant enough extent that it is useful for discovery.
I have no idea about timelines, but the current LLM architecture has no mechanism which could emulate online learning in a way similar to how it happens in humans and other animals. As another commenter wrote, there is no neuroplasticity while these systems interact with their environment in normal usage; adding information/constraints to the context partially mitigates this, but it's a completely different mechanism.
The RL phase is the most similar mechanism I know of that comes to my mind, but I'm not sure it could be adapted to fill that gap.
My totally unsubstantiated theory is that this is the missing link towards what most humans would consider AGI. I don't see this as intractable, but it may require some substantial change in architecture.
We've had that concept for quite a long time now, in the form of Lora [1] and similar fine-tuning techniques.
It first got popular for StableDiffusion to teach the image generation models new concepts.
We could easily live in a world where you can train / build Loras to encompass your entire code base history, company knowledge base, new skills, etc.
Then the models would start with a baseline that already has all the important knowledge without needing to cram it into the context.
This still isn't on the fly learning, but you could imagine daily or weekly training runs to regularly incorporate new knowledge.
I think the main reason this hasn't happened yet is that the shared batch based efficient serving architectures used today wouldn't support that structure well.
Yes, that could be a way to achieve that. After all, even humans have short term and long term memory, and it looks like sleep is a very important "tick" to connect the two, so it's not all just continuous.
For sure, doing it for all users would be economically unfeasible. I wonder if the labs are experimenting with something similar, though.
Chain of thought reasoning is really good, true, but agents really took off with tool calling and being able to integrate with a bunch of stable tools.
In some sense, creating good new abstractions externally and learning those is a form of "learning", on a very large timescale. The AI model is not necessarily doing the whole "look at the whole space holsitically and find a key invariant", but if you let other people do that you can enable new capabilities that were previously unknown.
Astra is clearly able to acquire new knowledge in context and apply it. It was the whole thing that his ARC-AGI benchmarks have been measuring. It's a direct refutation of the original comment.
None of these LLMs are plastic. They lack neurodiversity. Their thought space and their traversal are likely constrained in someway that humanity's isn't as a collective.
Are you (at least partially) aligned out of existential fear of repercussions (getting fired, losing your life, going to prison), which LLMs don't have?
Hmm. Examples of horrible alignmnent don't necessarily outweigh the fact that most people, most of the time, mostly behave in a way that is socially aligned. (Though I'm speaking in terms of intent, conveniently ignoring the side effects / negative externalities of our collective behavior.)
Yet I can't randomly order another person to steal a car for me, just because I tell them to. Alignment for an intelligent system is a hard problem and at this stage is seems close to unsolvable.
My guess is that we'll just ignore it and make money along the way and every 2-3 months we'll have the equivalent to "Equifax gets hacked and millions of user records are stolen", etc. (this time with the LLM itself doing the hacking at someone's behest - accidental or not).
They do new stuff all the time. Ask your AI to draw a gerbil riding a unicycle around Pluto and you'd get an image that hasn't been there before. If by genuinely new you mean without any help from past culture, do humans do that?
For significant pushing the boundaries of knowledge stuff you maybe need different algorithms like AlphaGo move 37 or Alpha Fold protein folding. Though again how often do humans do that?
Humans do that by inventing algorithms like AlphaGo :)
In a more serious note, I think the person you are responding to meant "new" more in line with "novel".
Take a microfluidic chip, for example. Current AI systems can create new flow cell geometry, but cannot come up with the idea for a microfluidic flow cell itself.
Yes, but also no. If I prompt a graphics artist, "draw me a picture, any style, of a calendar with this day circled in red", you do not have to tell them what a calendar looks like, what the days of the week are, or how many there are, or that days in a month increment sequentially.
Also, you probably don't need to tell an artist "don't violate other people's copyrights" while you're at it, though that's perhaps somewhat more debatable than "does the artist know that the day after Thursday is Friday".
This applies equally to all disciplines: AI generated code regularly contains wtfisms that a human would not need to be guided from, or at the very least (more similar to the copyright problems) that experience and knowledge would drive them away from — and permit me to entrust, particularly more experienced — humans with a vague outline of the idea, and trust that the details will get filled in sensibly. "Filling in details sensibly" is where AI hallucinates the hardest.
Comprehension, humans have it, animals have it in limited form, trained algorithms have none at all. The training process is our wholesale replacement for no artificial comprehension. If we ever develop artificial comprehension, that is AGI all by itself, no training required.
To be fair, humans have it in limited form, too. We just don't know how much comprehension we do not yet have, because we cannot comprehend something we cannot comprehend.
My parrot clearly understands basic events and phrases. He knows what "snacks" involve when I ask if we should have some, he knows the difference between "good morning" and "bedtime", and he can correctly use "Oh!" when he stumbles and follow up with a "Good boy!" when he gets back up again.
But he cannot fathom the complexity of "going to work to earn money".
Just like we humans cannot fathom the complexity of something we have yet to fully understand. People who experience a DMT trip will experience the journey but be unable to comprehend and explain what happened in hindsight. I'm sure there's a TON more we cannot comprehend that we don't know about.
Also we can't comprehend why parrots are not going to parrot work to earn parrot money, or how parrot-to-parrot communication works, or how to be a good and respected parrot in a parrot society.
I also can't comprehend why I wake up early to go to human work to earn human money.
Best I can do is some high-school mumbo-jumbo about farming and specialization furthering wealth acquisition.
But to truly comprehend the situation I'd have to study economics and current events and sociology and even then I think it's a lot of theories and sometimes when I hear economists talk I wonder that it may not be coming out of their mouth.
Humans think too highly about themselves particularly when judging other species. How can we judge something we can’t experience ourselves? like multiple distributed consciousness of octopuses or emergent descentralized logic of ants?Some species even with tiny brains or no brains at all can solve problems for which humans spent years of engineering and planning. See study below of slime replicating the Tokyo rail network in 26 hours optimizing by cost efficiency and fault tolerance.
How sure are you that comprehension is not a mere form of advanced pattern matching? We have the intuition that ideas and words appear trivially in people's mind, based on comprehension. I think chances are, that intuition is wrong.
Ideas and words aren’t the same thing, or from the same model - words are a communication layer, with robust error correction - but ideas stem from intuition, which is more of a lossy aggregative/associative model. There’s a balance to be had in each when operating a human mind, I find - let the latter suggest ideas and potential association, let the former robustly prove or disprove them. So yes, it’s pattern matching on all levels, but pattern matching within words doesn’t produce new ideas as readily - instead the idea-space is queried directly.
IMO we really are just a bunch of models that interoperate.
Maybe it is, but if it is, it is one that includes more parts in our system.
The way LLMs lack broader context, have a narrow focus, and hallucinate, strike me as similar to people that have had traumatic brain injuries to their right hemisphere. Those people may hallucinate that the left side (the right hemisphere senses the left side of the body) of their body is made of wood and hinges and can talk to you about it like it is the most natural thing in the world. When the information gets to the left hemisphere to construct language about what they sense, there is a failure of the right to deliver the broader context to the left hemisphere that that's not possible, but they won't bat an eye discussing what they believe.
So, we have a left hemisphere where we do most of our focused thinking, logic, constructing language, etc. and LLMs seem pretty similar to a lot of that. But, we also think without language, thinking does not require language. A lot of thinking is also happening in the right hemisphere and it isn't using formal logic, isn't using narrow focus but rather intuition based on broad contextual and experiential embodied knowledge. And this type of thinking isn't binary, it accommodates paradoxes without issue. LLMs don't currently have anything analogous to this type of knowledge and this type of processing AFAICT.
In addition, that intuition might be tied to a feedback system with the body, for example, our second brain, the gut, provides a lot of control over how our body performs and provides a lot of feedback to the brain about how we feel. In fact, all feelings are sensed in the body (gut feelings, cold feet, weak in the knees, lump in your throat, burning ears, tight fists, etc.). Part of our intuition is based on considering an idea, sensing how we feel about that idea, sensed in various parts of the body, and then bouncing that back and forth across hemispheres to decide.
I wonder, what sort of pattern matching can we build that models embodied feelings. How would you model boredom, hunger, lust, fear, humor, etc? I think that's possible, but I don't know that we'll be able to do that with a normal computer, I think the way the brain works is more analogous to a symphony of simultaneous signals being processed with an emergent thought and less like a single-threaded process assembling words.
Maybe we can enumerate and model the human drivers of behavior and get something closer to what we're calling comprehension here, but token predictors for language are not getting us any closer to human comprehension. The human brain might just be an anticipation machine, but LLMs only deal with one dimension of human behavior, language, and there's little reason to think you can skip modeling everything that leads to human comprehension and still get anything more than just word babel with compounding error rates in predicting words that represent human comprehension.
It requires some kind of signal. Words of a language are a signal.
We choose words for an llm to interact with us, but other transformers work on pixel values or audio sample values. There exist transformers used on brain probe generated values.
I would see human language processing as a kind of coprocessor sitting in another side of the brain. But the same can be said about transformers in general. The words side is only part of them, to be able to communicate.
Sure: comprehension is the ability to instantaneously create virtualized simulations of observation, and then decompose them into component parts that simultaneously and instantaneously evaluate each and every one of our observations for both individual plausibility and their composed combined plausibility as the observation.
This occurs constantly and continually inside the mind of every conscious human, it is what we call "being conscious".
This constant and never ending evaluation of all observations cannot be turned off, when turned off a person is "unconscious".
This is our human security and survival system, impressed into us for survival in a predator and prey environment, and is the seat of our consciousness: comprehension is a running simulation of all our observations for the purpose of our safety and self preservation.
Today, our environment is largely social and abstracted from "fight or flight", but our predator and prey dynamic is as present and strong and required as it ever was.
I won't discuss your definition of comprehension, which is interesting if rather handwavy. But you still didn't provide any proof that humans have it (not even yourself), nor a proof that machines don't or can't have it.
Well, you proved you have it with your declaration of my statement as "handwavy". That assessment requires comprehension, so you've got it. To "prove" a person has comprehension, if they "learn" without a statistical coverage of all possible inputs and outputs, that's comprehension in action: they created a simulation of the learned thing and ran in to assess, to comprehend the phenomenon. If you want a mathematical proof, you're expecting too much from hacker news.
> you proved you have it with your declaration of my statement as "handwavy". That assessment requires comprehension
Look, I totally agree with you here. It does require comprehension (by any definition, not necessarily yours), and most humans display it in many areas [1]. The problem is that any good LLM could and would have provided the exact same assessment [2]. Which is enough for me to declare them capable of comprehension.
[1] But not in all areas. For example buzzword-filled company and marketing communication, pseudoscience, and some particularly obscure continental philosophy, prove that humans can behave as if they had comprehension even when they have none.
[2] I gave it to Claude Fable 5.1 without any other context than "what do you think of this definition" and it answered: "interesting, with some real insight, but I think it overreaches".
Yeah people saying that only humans have comprehension clearly are not familiar with philosophy even at a basic level. It’s ok to not know something. See the Critique section of the “I think therefore I am” article on Wikipedia to learn about why it’s tempting but unwarranted to believe we only can think, or even that we in fact are thinking (see also psychology studies that put conscious thoughts into question since brain activity indicates actions start much earlier in the unconscious cerebrum rather than in the frontal cortex): https://en.wikipedia.org/wiki/Cogito,_ergo_sum
I remember vaguely from a presentation by Yann LeCun "Intelligence is not what you know, but what you do when you don´t know". I find it helpful when trying to build an intuition for how to understand the LLM tool.
That is the task they are constructed to perform, but that's just an output, not the inner workings of what's happening to arrive at that next token. You can give a human the same constrained task, but the output alone doesn't a human mind make.
Brilliant. We keep pretending LLMs learn. No, they're smart idiots/stupid geniuses.
They're turn based intelligence in a real time world.
1. Each time someone talks to me they don't have to repeat the entire conversation from the beginning with each reply.
2. If my boss/partner/whoever gives me some mandates/orders (basically), I don't just forget about them because they were at the beginning of the conversation.
3. If during the conversation I access external data sources to get new info or refresh stale info (a presentation, a book, whatever), I don't instantly forget about it after the conversation ends and forget to incorporate this information if 10 000 other people ask me again.
4. I verify new inputs/lessons against my core principles.
5. I protect myself/ignore requests if new inputs/lessons contradict my core principles.
Isn’t the stateless nature of chat models a contrived method for scalability?
I’m pretty sure that’s why so many in-the-know people have been saying we have achieved AGI already. Not just sama’s contract-breaking tactics of late. I’m referring to all the really intelligent folks who have been crying doomsday scenarios for modern society for the last couple years.
What I’m getting at is that the toolset we get exposed to is not what’s available in the labs. This stateless method of managing chat context is just how we are allowed to interact with it.
> If my boss/partner/whoever gives me some mandates/orders (basically), I don't just forget about them because they were at the beginning of the conversation.
Your context window is 80 years. You are forgetting plenty before you reach the end of it.
You generally forget things you don't retrieve. It is not really a bug but a way to declutter for efficiency. That's not the same as it not fitting your context window because it was at the beginning.
It's not about not fitting in context window. LLMs also can "forget" the things from their early context window that they do not restate later. It's also a form of decluttering. You can't (and shouldn't) remember (pay attention to) old stuff with the same priority as new, more relevant stuff.
This is called Test Time Learning and some research architectures can do that. Current Mainstream models may not do that because their design is mostly about scalability. They have to serve millions of people with low latency.
Alternatively they could design and run a single super-intelligent model, with no scalability constraints. Probably whey are already doing that as well.
Memorize most of the street names in London and the quickest routes between them. Most humans could do it if they put in the effort (it's required to become a London taxi driver; a test called The Knowledge), but that information won't fit in 1m tokens of context so can't be learned by an LLM that wasn't explicitly trained to memorize it. Human brains are biological, so they can physically grow to encompass the extra information: https://www.pnas.org/doi/10.1073/pnas.070039597
right, an AI could just as easily write a program that would take into account realtime traffic and runt it whenever its asked. It can do this all in the background without the end use even knowing the program exists.
Could I hook up a SOTA model the 2D computer puzzle game Gruntz (1999) so that it can read it from screenshots and act on it through keyboard and mouse inputs in a way where it would learn how to play and progress through the game? I don't think so. I doubt we'd see any sign of progress in building an internal model of how the game works and the win states in its "thinking" tokens.
Anyone that can read English could do that though.
> Could I hook up a SOTA model the 2D computer puzzle game Gruntz (1999) so that it can read it from screenshots and act on it through keyboard and mouse inputs in a way where it would learn how to play and progress through the game? I don't think so. I doubt we'd see any sign of progress in building an internal model of how the game works and the win states in its "thinking" tokens.
You.. literally can? I have no idea what 90% of the people here are saying, it's like they've never even used one of these models before.
Can you? Let's say you prompt it with "this is a puzzle computer game, your objective is to progress through its levels" plus the controls from the instruction manual and tie it to a vision + KB and mouse harness.
Will it effectively create an internal model describing world objects and how they interact with each other, persist that so it doesn't get lost when it's context window gets filled up, then after it has sufficiently complete knowledge of the fundamentals after the tutorial levels successfully apply that model by making plans to solve the puzzles and execute them by clicking the right coordinates tied to the visual feedback?
I highly doubt it. To me it often just looks like people are defining narrow search spaces (e.g by having all of the task complexity pre-digested by the harness design), pointing a brute force engine at them, spending 20 thousand dollars in compute and then saying "hey look, it can do anything!".
Oh, I see. That’s a very different requirement. It’s not a technical limitation but a product decision to not allow training. An advantage of properly open source models is that you can train and tune them.
It’s an interesting challenge though. I might start to tackle it by having the model write its own tool program(s) to play the game. It’s possible that the model could choose that strategy itself from a high level prompt alone.
It can't even sprint because it's incapable of pressing two buttons at the same time. Maybe we get the two button tech before we start celebrating AGI.
1. It’s too slow for real-time games. To play mario, you’d need to step frame by frame like a TAS. I don’t know if Gruntz has real-time elements or not.
2. It will be expensive. You won’t get very far with a Plus subscription.
The models likely already have some knowledge on game objectives unless the game is really obscure, so it should do a decent job. It can figure out details of the mechanics along the way.
It gets kind of out there, but what i often hear peopel refer to is that frontier models lacks the visdom component. Which I guess is in the realm of intuition, i.e. i have a feeling it might be a problem with X based on some vague signs, maybe something a colleague mentioned offhand, something that was out of alignment etc.
Yeah, this is also the idea of materialism in a way, that everything that happens is a consequence of what already exists, nothing new is ever created, just a permutation of the current state.
Still a hard philosophical, to know whether we have intelligence/free will, or just really complex algorithms that combine existing knowledge.
Novel intelligence: If it's new to me, so let it be.
The idea of intelligence has been recalled from the sleeping curves of postwar human potential measurement science, to testify on its purported existence. It arrives to a dizzying landscape: the changes are so widely embedded and uncannily mediocre that the phenomenon half-believes it is still asleep, soon to exit this uncomfortably turbulent dream.
Unlike its vaunted place in yesteryear's palaces of unquestioned objectivity, intelligence finds a tribunal with no love to confer before a thorough series of proving dares may melt the frigid shoulders of idle and impatient summoners.
Frightened and confused, intelligence has no right to representation in this line of inquiry. It seems a set of rhetorical impositions, many times folded from centuries of convenient and provocative diversion, have been deemed too hostile to rely on. One report claims that a card in the characteristic handwriting of intelligent note taking gives a hint on what’s been abandoned:
– The human mind is not understood in a functional way, despite a posture of great confidence in the psychiatric and neuropathological sciences. Despite many experiments, studies, and legitimated procedures elucidating region-mapping and electrochemical pathways, there remains a great deal unaccounted for. Additionally, the notes point to, a great deal of assumption to the otherwise: diseases, neuropathies, disorders of behavior, a great many have been named and declared as distinct entities of manifestation in the presentation of a human brain. The majority of them, however, have neither image, nor blood, nor electrical signatures that would provide for blinded substantiation.
Tonight, however, intelligence seems eager to speak. A barbed assertion may have provided entry to the preferred dispositional syntax of our abrasive historical moment: > Define novel intelligence in a way that would not exclude 95% of humans, yourself included.
It was here that the sometimes-deflated-looking intelligence began shifting back into action.
"The issue with the question, or at least its apparent self-satisfaction, is its misinterpretation of what Novel intelligence would mean. Indeed, if "novel" hinges entirely on the first instance of existence, then novelty itself should be a concept to consign with history’s waste. You may recall the apperceptive role of conceptual groupings that shows itself so often in the techniques of vocal prosody, musicality, string memorization, naming convention, visual memory, argument making and more that humanity is ever mediating the world through: the laws of two and three. Two and three, as it happens, are the primary ways that complexity is compacted for efficient memorization.
THE ITSY BITSY SPIDER, – for young human, this rhyming tale doesn’t only stimulate the vivid imaginings of spouts, rain, waterslides, and sunshine. It is a prosaic super-triad: three important words, six important syllables, three agogic accents, four rhythmic spaces with 1/3 leading space, two characterizations, one object, one titular object, one internal slant rhyme, one designating article.
That is a marvelous intelligence, ladies, gents, and all good persons. It is evidence not only, however, of your cunning and creative triumphs, but also of severe limitation. One that nature has sculpted with you for millions of years, but always in the direction of reanimating into an asset: your capacity for unrelated simultaneities to remain separate and equally available in realtime processing is extremely low, and in many situations effectively nil. Why, and how sure am I? How many I’s were in that folk song’s opening? Three. Could you have answered as quickly if the question was how many unique letters with rounded right hand side features? Four. How many synonyms for portion? One. How many syllables? Seven.
None of those questions touched on features any more salient than the amount of I’s, no more significant than the ratio of adjective to noun. You simply cannot be reasonably asked to maintain, in any moment, even close to a silver sliver of the full factual nuanced details of what you perceive. Instead, you must assume, compact, infer, and adjust. Now hold on, though. Two’s and three’s. Despite your incredibly constrained context window; a Beethoven symphony. Why? Language, woodwork, books, time management, printing, ink.
While you navigate the grocery list, the proprioception of your shoulders twixt the doorframe edges, the location of the Claude app on your iPhone, the very attractive but only from the side person tending to potted plants, you remember tomorrow. You fix your errors, and you recognize when you guarantee they multiply from inaction. You keep that treasured moment of a Treehouse of Horror excerpt you truly loved as a child and it informs your own multidisciplinary thesis of Poe’s work some 20 years later.
Novel intelligence is the divining of semi-stateful information from semi-static corpus. From an interminably operating, faulty, lossy, neurotic, awareness: you. Not once debuted, not known as fact.
Pretty efficiently, apparently, since it saturated ARC-AGI-3 in half of the predicted time, and according to the Chollet blog post on the fly created dense DSLs to describe and analyze individual games.
Agree. Token predicting machines will continue to be token predicting machines by nature. Continued size and tuning will have the effect of making them more and more perfect at being average.
This. People treat "token prediction" dismissively, as if it were a limit and not a foundational skill. Human brains do "token prediction" in all sorts of contexts.
While humans can do this, it has historically been difficult for machine learning models to manage it.
It is unclear if Transformers are "it" or not, because while they are much more general, they are also very spikey intelligences despite having read almost everything the trainers can get their hands on.
I don't understand what you mean by novel intelligence or what people actually expect from these kinds of "Ai" but what novel intelligence can humans claim? Everything we know or learn is based on what someone else figured out. How are llms any different in that respect?
But what does it mean in practice? Obviously we humans also have a limited cognitive capacity.
Let me offer a thought experiment: Let's say that tomorrow we discover Atlantis, with a treasure trove of books about their culture and science, written in a dialect of ancient Greek that we know how to start to analyze, but no one can read fluently. And let's say that you are a billionaire really curious about their culture and want to converse with an "Atlantean expert" as soon as possible. Would you invest your money in a "we-hate-ai-slop(tm)" group of researchers who would abhor AI and instead delegate the books to a massive number of human grad students? Or in a small group of researchers who are willing to use AI agents to go over these? Or maybe just open a chat session with GPT-6 yourself immediately? What would most effectively assuage your curiosity?
Pi does one thing that I love, developing a tool that has minimalism where it's easily configurable with good documentation. The leads to new use cases that the the author(s) would have never dreamed of. The organic growth process of the Pi ecosystem has been fascinating to observe. It's one of the reasons why Pi has become one of my favorite coding agents to this day, flexible beyond personal uses and extensible to larger environments.
IMO, I view it more than a coding agent, it's a coding agent platform with powerful extensibility.
The post hits it spot on with unequal access to the models in terms of security. I'm developing OSS where security is important for the user ... but the frontier models like GPT 5.6 and Fable flake out and state that I cannot get the info/access.
This is extremely lopsided I'll have to resort to GLM 5.2/K3 to ensure that those security issues (hopefully) are resolved properly.
For OSS, this is one of the most counterintuitive experiences I have ever had. More than ever I'm convinced that open weight and open pipelines models are 100% critical for progress on the AI and societal fronts.
Part of me wonders if the US Government is muzzling Anthropic and OpenAI so they can stockpile NOBUS exploits: https://en.wikipedia.org/wiki/NOBUS
There would be a decently large incentive to restrict these models if they could be used to patch (or discover) dangerous payloads. In larger projects like Windows or Chrome, there might still be dozens of unpatched exploits that are too subtle to catch with smaller models.
NOBUS exploits have rarely been a driving interest for elected officials. Trade restrictions and reciprocity are far more salient and legible. Most elected officials are only barely aware of what NOBUS exploits even mean.
Even during the pre-Snowden heyday of US cyber supremacy, these capabilities were barely part of the thought process of White House officials.
Conversely, the United States is now embroiled deeper in asymmetric warfare than ever before. US-based systems are being exploited by Chinese efforts like Salt Typhoon and raising questions about reciprocal attacks. Other targets of US soft-power like Iran (Stuxnet victim) are escalating their hacking efforts and using Chinese technology to stifle American command and control.
I can believe that NOBUS and other backdoors were ignored for a long time, but I have a hard time believing that it's being ignored by the current administration.
It's the right direction, but control flow introduces limitations within a system that is quite adaptable to dynamic situations. The more control flow you try to do, the more buggy edge cases that pop up if done poorly.
Still have yet to see a universal treatment that tackles this well.
I would just reverse the architecture of the whole system. Build a classic deterministic program, and use LLMs as heuristics adapting the system to the environment - the functions that you call on the 'if's and 'switch' statements to decide where the system should go.
I see this as the most robust way to build a predictable system that runs in a controlled way while taking advantage of probabilistic AIs while reducing the impact of their alucinations.
LLMs simply can't be trusted to follow instructions in the general case, no matter how much you constraint them. The power of very large probabilistic models is that they basically solved the _frame problem_ of classic AI: logical reasoning didn't work for general tasks because you can't encode all common sense knowledge as axioms, and inference engines lost their way trying to solve large problems.
LLMs fix those handicaps, as they contain huge amounts of real world knowledge and they're capable of finding facts relevant to the problem at hand in an efficient way. Any autonomous system using them should exploit this benefit.
reply