yes, but for you all of that text is associated with ideas. The word "dog" has an associated object. For a machine like GPT-4, the word "dog" has no meaning or object, but it does have an associated likelihood for adjacent words. The words themselves aren't the intelligence, the ideas behind them are.
It's an open question the extent to which language models can learn the underlying meanings of words.
There was a recent podcast with Sean Carroll interviewing Raphaël Millière where they go into this topic and some of the research on it. Two examples I can remember are: 1) DALL-E had subject-specific, domain-specific neurons, 2) language models' representations of color terms encoded the geometry of the underlying color space, e.g. vectors in RGB space.
I don't think we should be too quick to assume how these models work. There's a lot that even the researchers don't know and these are empirical questions that can be studied.
Before all this Machine Learning madness, while I was studying unsupervised learning I had the following thought:
To cluster is to create an ideia, a concept which encompasses a set of data points. A cluster might be bigger than the set of points with which was created and intersect with other clusters. This is still what I visualize when I see image models interpolating between "concepts".
Believing that all an LLM does is the likelihood of adjacent words is probably an oversimplification. My verdict about this topic is: I don't know and I'm totally confused about what interpolation and extrapolation mean in higher dimensions
This sparks something I have always been fascinated by, hopefully someone can illuminate this for me.
If someone was born with profound blindness, or deafness or any profound lack of sense - what do they see in their heads? Or hear? I hear a version of my voice in my head, but some people have no inner dialogue at all, or even no ability for inner imagery.
Perhaps, but i think there's just as much hubris in saying "We've created something smarter then us" as there is in saying "We're smarter then the thing we've created".
meanwhile, we've created the worlds most complicated set of dominos, and we're delighting in knocking them over.
I want you to only use English words of Germanic origin; avoid using words derived from Latin or Greek. If there's no alternative, make up a new word by calquing the original into Germanic roots.
--- output ---
The sunstead is the sun and all things that go around it. In the middle is the Sun, a big, hot ball of glowing gas. Around the Sun are eight big worldrocks. From nearest to farthest, they are:
Mirkworld - The nearest worldrock to the Sun, small and fast.
Lovegoddessworld - The hottest worldrock, with thick air and no water.
Earth - Our home, the only worldrock known to have life.
Redworld - A red, rocky worldrock with thin air and high mounts.
Greatthundergodworld - The biggest worldrock, with many fellowtravellers and a big red spot.
Harvestgodworld - A gas worldrock with a thin ringstead around it.
Skygodworld - An iceworldrock that lies on its side, with a thin ringstead.
Seagodworld - The farthest iceworldrock, with strong winds and blue clouds.
Besides the worldrocks, there are smaller rockbits and icebits that also go around the Sun. The rockbits are mostly in the Rockbitbelt between the Redworld and Greatthundergodworld. The icebits come from the far edges of the sunstead, in the long way out Icyrockcloud and the faraway Cloud of All.
The sunstead has a flat shape, with most things going around the Sun on near level paths. The worldrocks go around the Sun in a way called nearround paths. The Sun's pull keeps everything in the sunstead together.
Beyond the worldrocks, rockbits, and icebits, the sunstead also has spacegrit, small bits of rock and metal that fly through space. When spacegrit enters Earth's air and burns up, it makes a lightstreak that we can see in the night sky. If a bit of spacegrit reaches Earth's ground, it is called a starstone.
The Sun, which makes up most of the sunstead's weight, sends out a flow of charged bits that spreads far into space. This solar wind forms a wide bubble, the Sunshield, that surrounds the sunstead and helps shield it from high-speed bits coming from outside.
The sunstead's workings are steered by the pull between the Sun and the things that go around it. The worldrocks and fellowtravellers tug on each other, leading to small changes in their paths over time. The Sun's pull is strongest near it, so worldrocks closer to the Sun go around faster than those farther out.
As the Sun ages, it will go through changes that will have big outcomes for the sunstead. In about 5 billion years, the Sun will bloat into a Red Greatstar, swallowing Mirkworld, Lovegoddessworld, and maybe even Earth. Afterward, the Sun will shed its outer layers and shrink into a White Dwarf, a small, hot, dim star.
Lastly, the sunstead is but one of many star systems in our Starset, the Milky Road. The Milky Road is a wide, spiraling set of billions of stars, gas, and dust, and it is just one of countless starsets that make up the great Allthing.
I would've like that chatgpt came up with better names for the planets Venus, Jupiter, Saturn, Uranus and Neptune. Mirkworld, Earth and Redworld were just fine.
The following is an actual transcript of a conversation with GPT-4.
--- input ---
I want you to replace the word "right" in your output thereafter as follows:
if it indicates direction, say "durgh;
if it indicates being near or close, say "nolpi";
if it indicates correctness, say "ceza".
I will also use these replacement words accordingly and expect you to be able to understand them.
Write a 20-sentence story that showcases these rules. It must contain no word "right", but each replacement word must be used at least 3 times in a proper context. You must not use the replacement words for names and other things unrelated to their meaning.
--- output ---
Once upon a time, in a small village nestled deep in the forest, lived a young boy named Timmy. He was a curious and adventurous child, always exploring the durgh side of the woods, despite his mother's warnings. One sunny day, as Timmy ventured durghward, he stumbled upon a mysterious cave.
Feeling a mix of excitement and fear, Timmy hesitated for a moment before deciding to enter the cave. As he took his first step inside, a soft voice echoed from the nolpi darkness. "You have entered my domain, young one," said the voice. Timmy was surprised but felt a strange sense of ceza in the voice, so he continued deeper into the cave.
The walls of the cave were adorned with ancient drawings that seemed to tell a story of a time long past. Timmy noticed that some of the drawings were marked with a peculiar symbol, which he realized indicated the durgh direction. Intrigued, he followed the symbols, each one leading him nolpi to the heart of the cave.
Finally, Timmy arrived at a massive chamber with a shimmering pool of water in its center. Standing beside the pool was an elderly woman, who introduced herself as the guardian of the cave. "You have come a long way, young one," she said with a gentle smile. "I can see the ceza in your heart, so I will grant you one wish."
Timmy thought hard about what he wanted most in the world. He knew that his village was suffering from a terrible drought, and their crops were withering away. So, he wished for rain to fall upon his village and restore their crops to their former glory. The guardian nodded in ceza, and the pool of water began to ripple.
As Timmy left the cave and ventured back towards his village, he noticed dark clouds forming in the sky. Soon, a gentle rain began to fall, and Timmy could see the ceza in his decision. The villagers rejoiced as their crops flourished once again, and life in the village returned to normal.
However, Timmy knew that his adventure was far from over. The guardian of the cave had shown him that there was much more to the world than what lay on the durgh side of the woods. He vowed to explore even further and discover the secrets that the world held.
With a newfound sense of purpose, Timmy embarked on countless adventures, always remembering the lessons he had learned in the cave. He knew that by following his heart and trusting in his instincts, he would always find ceza in his decisions.
Throughout his journeys, Timmy became a legend in his own durgh, and his name was spoken with reverence by those who heard his tale. And although he traveled far and wide, he always returned to the village that he loved so dearly, never forgetting the friends and family who were always nolpi to his heart.
---
As you can see, it's perfectly capable of using invented words - which are not a part of the training set, and thus have no "associated likelihood" of any adjacent words - according to their designated meaning, i.e. "the ideas that they represent). It does occasionally make mistakes, but way fewer than if it were using them randomly (this is easy to see by using a less powerful model - e.g. GPT-3.5 replacements are mostly random). Not only that, but if you ask it to double-check the text it wrote without pointing out specific mistakes, it can usually spot them.
