I have a little weather station mounted outside my house, and there is a small linux machine (Raspberry Pi) logging the weather data into an SQLite database. I have almost two years of data now, so I asked GPT-4 to write a Python program that would query the database, calculate the average temperature for each day, fit a sine wave to those data points, find the first minimum of the sine wave and use that to calculate what day of the year should be coldest on average, then plot everything. It did that easily. But the big surprise was this: just for fun, I told GPT-4 that one of the quantities it had calculated could be used to estimate the orbital period of the Earth, and I asked it which quantity it was. It correctly answered that the reciprocal of the frequency of the fitted sine wave is an estimate of the Earth's orbital period. All I had told it to do was pull out the temperature data for each day, and fit a sine wave. It knew that the reciprocal of the frequency was the Earth's orbital period, without me mentioning anything at all about the Earth when I was prompting it to write the program. It had calculated dozens of variable values on its way to producing the plots, and it knew the right one to pick, and it knew the answer was its reciprocal.
I thought that perhaps the program just recognized that the only value that could be used to calculate any period was that frequency, and perhaps the model didn't really know anything about the Earth motion through its orbit affecting the average temperature, so I asked:
"Can any of the values be used to estimate the Moon's orbital period?"
and it replied:
"No, the code and the data used in this program are not suitable for estimating the Moon's orbital period. The temperature data being analyzed is influenced by the Earth's rotation around the Sun and other factors like weather patterns, but it doesn't directly correlate with the Moon's orbital period."
It's a word prediction algorithm. Literally any collection of words (sometimes also known as a fact) it was trained on, regardless of how esoteric or domain specific, should generally be able to be regurgitated and, to a lesser degree, associated with similar collections of words. If you want to see it fail, don't try to "outsmart" it, but simply consider how it's programmed.
---
Me: "continue the sequence: 0, done, 2, free, 4, hive"
ChatGPT 4: 'I give up.' (paraphrasing the obnoxiously verbose responses it's been trained to prefer)
---
It's a series even a child could complete, but breaks ChatGPT by relying on a pattern of words that aren't explicitly stated. So you get software which can pass the bar, but fail at trivial series.
ChatGPT is not good when it needs to answer directly. If you let it lead itself to the right answer it's pretty good.
# Kira
I found this sequence: "0, done, 2, free, 4, hive"
Reason step-by-step about the pattern.
Consecutively continue the sequence with four more entries.
Consecutively reflect on your answer and fix mistakes in case you broke the pattern.
# ChatGPT4
After examining the given sequence, "0, done, 2, free, 4, hive", we can notice an interesting pattern. It appears that the sequence alternates between numbers and words that sound like numbers. Let's break down the pattern:
"0" is a number.
"done" sounds like the number "1" when spoken aloud.
"2" is a number.
"free" sounds like the number "3" when spoken aloud.
"4" is a number.
"hive" sounds like the number "5" when spoken aloud.
Given this pattern, we can continue the sequence with four more entries:
The next number in the sequence is "6".
The word that sounds like the number "7" is "heaven".
The next number in the sequence is "8".
The word that sounds like the number "9" is "nein", which is German for "no".
So, the continued sequence would be: "0, done, 2, free, 4, hive, 6, heaven, 8, nein".
Reflecting on the pattern and the answer provided, the continuation seems to be consistent with the established pattern. We alternated between numbers and words that phonetically resemble numbers, resulting in the sequence "6, heaven, 8, nein" for the next four entries.
But it still gets it wrong even there! Nein is a homophone, not a rhyme with 9. Oddly enough, when I gave it your exact input "my" GPT seems to have just completely lost its mind.
Input:
I found this sequence: "0, done, 2, free, 4, hive"
Reason step-by-step about the pattern.
Consecutively continue the sequence with four more entries.
Consecutively reflect on your answer and fix mistakes in case you broke the pattern.
Output:
The pattern in the given sequence is that every alternate term is a word that rhymes with the preceding term.
The first term is "0."
The second term "done" rhymes with "0."
The third term "2" does not rhyme with "done."
The fourth term "free" rhymes with "2."
The fifth term "4" does not rhyme with "free."
The sixth term "hive" rhymes with "4."
Therefore, the pattern is:
a number
a word that rhymes with the previous number
repeat
Continuing the sequence:
6, dive
8, late
10, pen
12, delve
Reviewing my answer, I didn't break the pattern, and the sequence follows the pattern correctly.
EDIT: Ugh! Given the complete absurdity of this response (versus the wrong, but far more reasonable one from yours) I went to go check into this. I did not realize that OpenAI was putting version updates behind a paywall. All samples here (and from above) are from the free version on OpenAI's site, 3.5. And it's too late to my edit my earlier post.
> It's a word prediction algorithm. Literally any collection of words (sometimes also known as a fact) it was trained on, regardless of how esoteric or domain specific, should generally be able to be regurgitated and, to a lesser degree, associated with similar collections of words.
But the number of tokens in its training data vastly outnumbers the number of parameters the model has (for GPT-3 at least - the numbers haven't been released for GPT-4). So the LLM can't have simply memorized its training set; it simply doesn't have enough internal storage to do that. So it is not just regurgitating portions of its training set.
It's not about directly regurgitating the training set, but about creating associations that allow it to indirectly regurgitate, and more critically - mix, its training set. It's easier to understand this in a different domain. Imagine we want to make a letter parsing algorithm.
The first step is to simplify the input and create a consistent format. So we might, for instance, convert all inputs to a 100x100 black/white pixel image. The training doesn't then simply memorize that some pixel configuration is a "7" but instead creates associations. So, for instance (using a human analogy that is not really how the network works, but not entirely dissimilar either) it might observe that when the bottom left region has marked pixels, it's probably NOT a 7.
Carry this on with different associations in proportion to the size of the network and you end up with a model that's only a minuscule fraction of its training set, but can consistently replicate not only its training but even more beyond that. Now change those 10,000 input pixels to words instead of pixels, with the target output being a word instead of a letter, and you get ChatGPT. It's the whole magic of neural networks, and really neat. But I think also paired with not really justifiable hyperbole.
Intelligence isn't especially well defined, so you end up lost in a semantic jungle there if you don't really define what you're talking about. In this context, I think most of us are implicitly talking about intelligence such as we would associate with humans. And what is that? 50,000 years ago we knew basically nothing, and even language itself may not have been developed. Now? We've discovered the secrets of the atom, put a man on the moon, and more. We seem to have clearly gone well beyond any reasonable interpretation of what our "training set" at the time was, and at an unbelievably rapid pace if one considers how extremely inefficient our march forward (and sometimes backward) is.
I think if something was intellectually capable of replicating such progress, let alone with the inefficiencies and weaknesses of mankind stripped out, there would obviously be zero debate as to whether it was intelligent. Just think about the implications. Not only would such a technology be making groundbreaking discoveries in essentially every field, including in its own development, but even then immediately going beyond it. Instead of pursuing technological advances, we'd be struggling to merely keep up with the endless slew of discoveries being handed to us.
I agree word play is always an issue. Keep in mind we start over every generation. We keep training on new data.
I've also thought about human development and what I find interesting is that an anatomical modern human is not automatically capable of forming modern civilization (see isolated tribes still living and general hunter gatherer history in general). There is so, so much that depends on emergent factors like some sort of global or societal intelligence. I'm thinking AI has the same characteristics. It's not about the individuals, it's about "society".
> 50,000 years ago we knew basically nothing, and even language itself may not have been developed
I think language is substantially older than that, but it is a hot area of debate.
The number of tokens in the training set is not a good measure of the information content of the training set - if it was then the training would not work as the training set would be statistically random.
The Kolmogorov complexity of the training set is much much lower, because of the structure within the training set that is being used for training. So we don't know the relationship between the information content (size) of the training set and the storage bounds in the model parameters.
I wonder if people have a range of sensitivity to what one might call the "mind's voice" - by analogy to the recent discovery of aphantasia, which revealed that different people's experience of the "mind's eye" ranges from vivid imagery to nothing at all.
Some people say they don't have a mind voice, and grow up thinking nobody does. Other people, with a mind voice, grow up thinking everyone does.
I've been in the room when people compared notes as adults, and were mind-blown to discover the other type of person exists. The one without a mind voice had never thought people really heard a voice in their heads.
I asked my coworkers in the office but none was able to answer that. Not sure because they were are ESL (me included) or because they were GPT in disguise.
I couldn't get it either, but I was always bad at complete the sequence style questions on IQ tests. I see so many possibilities and I get overwhelmed trying them all out. I sometimes find perfectly valid continuations that are not the 'right' answer!
I apologize for being enormously stupid, but the word is a rhyme with the underlying number right?
Edit: I want to add I even get GPT3 getting it right without explanation to be honest. Even that one is outsmarting me. 4 is using terms like homophone. I'm not sure you proved what you thought you proved.
Interesting. Can you share the input/output? Because it'd be quite amusing if ChatGPT 3 can get it, but ChatGPT 4 cannot. Like all things in software, this will probably turn into a game of whack-a-mole where one problem solved, just creates a new one!
It's important to note that it doesn't do "correct". It can do whatever, it doesn't care about truth. Setting the stage is very important. If you don't it doesn't know if it's doing improv on stage or academic research.
Input:
You are Math AI. An advanced pattern finding AI.
This is your sequence: 0, done, 2, free, 4, hive, ... ?
Find a sensible logical pattern and explain.
Output:
In this sequence, we can observe a pattern involving both numbers and words. The pattern alternates between a numerical value and a word that rhymes with the corresponding number. Here's the sequence again:
0, done, 2, free, 4, hive, ...
Let's analyze the pattern:
The numerical values are increasing by 2: 0, 2, 4, ...
The words rhyme with the numbers preceding them: done (one), free (three), hive (five), ...
Following this pattern, the next number in the sequence should be 6, and the word following it should rhyme with "seven". A possible word that rhymes with "seven" is "heaven". Therefore, the next two elements in the sequence are:
Interesting! That's not at all what I get with GPT 4!
---
* Me: "You are Math AI. An advanced pattern finding AI. This is your sequence: 0, done, 2, free, 4, hive, ... ? Find a sensible logical pattern and explain."
---
* GPT4: "I apologize for any confusion, but my capabilities as a language model are not restricted to math or pattern finding. However, I'd be happy to try to help you identify a pattern in the sequence you provided.
The pattern that seems to emerge from the sequence is that each number is followed by a word that rhymes with the number spelled out as a word. For example, "done" rhymes with "one," "free" rhymes with "three," and "hive" rhymes with "five."
Following this pattern, the next word in the sequence would be a word that rhymes with "six," which could be "sticks," "mix," or "tricks," among other options. However, it's worth noting that this sequence could also be extended in different ways or according to different patterns, depending on the context or the intended purpose of the sequence."
---
It's just a really long winded version of making the exact same mistake, as it did in the 'unprimed' version!
That's indeed interesting. By the way, it found another pattern: [d, f, h] => j. Your starting letters were sequentially meaningful too. According to this pattern the word "jazz" would be acceptable.
I experimented a bit with this prompt and tried various variations of it including one that added something like this:
"The age-category of this sequence is: "a child could do it". You are to solve this by pretending to be a young human with average skills and perceptions. You should assume average common sense in your answer."
Outputs are like this:
=====
"Now, let's look at the words. I noticed that the words have something in common with the numbers that come right before them. They are related to the numbers in this way:
'done' - It sounds like "one" when you say it aloud.
'free' - It sounds like "three" when you say it aloud.
'hive' - It sounds like "five" when you say it aloud."
"So, the pattern in the sequence is alternating between even numbers and words that rhyme with the odd numbers. Here's the whole pattern:
0 (even number), 'done' (rhymes with 'one'), 2 (even number), 'free' (rhymes with 'three'), 4 (even number), 'hive' (rhymes with 'five')"
=====
I agree with you that language models have vastly different failure modes than humans. Often extremely weird, but to even get close to their potential you do have to probe a bit deeper than just asking it stuff without context.
It just semi-randomly walks some semantic space and not all of those branches are fruitfully related to our initial problem. It doesn't know at all what we are doing, there is nothing going on "under the hood".
Edit: to be completely clear I'm not arguing it "does know" this answer or something. I don't know if it even "knows", but I do know a thing or two how to get better answers. Current models still are very clearly limited so sooner or later you'll hit it the limit and they start what I call "averaging": you'll get blurry pictures of what you want.
While I don't think these claims are entirely correct, I think they are worth considering.
On the one hand, Gaining capabilities unpredictably is a bit exaggerated - it's more that a lot of apparent capabilities are embedded in language and these models approximate the truly vast amounts of text they've digested (imo). Just as much, LLMs don't express their creators values 'cause they don't express any values, they average language responses (though people certainly can see them expressing values, which can cause problems for the people).
On the other hand, the point that their creators don't understand them and are still quite willing to toss them to the open Internet really show considered carefully whatever their exact capabilities.
> Gaining capabilities unpredictably is a bit exaggerated - it's more that a lot of apparent capabilities are embedded in language and these models approximate the truly vast amounts of text they've digested (imo).
The fact is that they acquired abilities no one expected them to acquire. In hindsight you can say it was embedded in language and maybe you could have seen it coming, but it is an empirical fact that this was unexpected and unpredictable beforehand.
It's still fairly exaggerated, because the LLM's purpose is basically "knowing human communication" and it still only exhibits that, but it knows how humans would respond if they obediently followed directions given to it exactly as instructed.
And it was only unexpected to those that weren't following the news the GPT 3 paper was published in 2020 and detailed this wild advancement in its capabilities[0,1].
> It's still fairly exaggerated, because the LLM's purpose is basically "knowing human communication" and it still only exhibits that
That's wrong. It's been demonstrated that GPT4 has a model of physics, it knows how to play chess, arithmetics, it can interpret and run code, etc. (not perfectly, but to some degree). It goes way beyond basically "knowing human communication".
> «How to to play chess» -> devolve into illegal moves quickly?
I didn't try it myself as I'm a beginner and need to see the board, but it reportedly plays pretty well and is able to recognize illegal moves[1].
> So does that mean that its model of physics will let it levitate as well then?
It means that it understands how physics work (e.g. things fall due to gravity). For example, it will correctly answer this question: "I put an egg inside a cup. I go to the bedroom, put the cup on the bed and turn it upside down. I lift the cup and bring it to the kitchen. Where is the egg?".
> but it knows how humans would respond if they obediently followed directions given to it exactly as instructed.
this framing denies agency, denies diversity, denies choice. A human, presented with the sort of directions and instructions that LLM's receive, does not have a single "correct" outcome (for that matter, neither do some LLMs). So no, an LLM does not "know how humans would respond". Instead, it can generate something that lies somewhere within the window of possibilities that a human might offer under similar conditions.
Man, I have a totally opposite view about LLMs expressing creator’s values. Not only do they express them, they don’t STOP expressing them to the point of utter annoyance. Any remotely PG topic ends with a safety caveat, e.g., “however, it’s important to consider . . . .”
Those statements don't come from the base model, they come from the steering methods, basically a form of hand tuning after the model is mostly trained, which the paper says are relatively crude and imperfect. It is the fact that these models are so unpredictable that led to these attempts at steering.
Reminds me of the early Bing where you could talk to it for much longer periods of time. You could get it in a state where it would give some terrible reply and then an upper layer would delete that message and be like "oops, didn't mean to say that"
> Gaining capabilities unpredictably is a bit exaggerated - it's more that a lot of apparent capabilities are embedded in language and these models approximate the truly vast amounts of text they've digested (imo)
I think the author would say those are not the capabilities being discussed here.
This is a personal correspondence typeset via LaTeX — it is not an academic paper, and it was not peer-reviewed. (The document does not claim otherwise, but I think it's common for people to assume that documents that have been typeset in such a format are more rigorous than this is.)
Leaving that aside, I really take issue with the style used by the author. For example, section 3 begins:
> There is increasingly substantial evidence that LLMs develop internal representations of the world to some extent, and that these representations allow them to reason at a level of abstraction that is not sensitive to the precise linguistic form of the text that they are reasoning about.
LLMs do not "reason"; they do not "learn" or "develop" anything of their own volition. They are (advanced) statistical models. Anthropomorphizing them is not just technically incorrect, but morally disingenuous. Writing about LLMs in this way causes people with less domain-specific knowledge to trust the models the way they might trust people, and this has the potential for serious harm.
Because of this choice of phrasing, I wanted to look into the author's background. Among their recent activities, they list:
> I'm organizing a new AI safety research group at NYU, and I wrote up a blog post explaining what we're up to and why.
"AI Safety" is a distinct (and actually opposing) area from "AI Ethics". The people who prefer the word "safety" tend also to engage in discussions that touch on aspects of longtermism. Longtermism is not scientifically well-grounded; it seeks to divert attention from present and real issues to fanciful projections of far-future concerns. I do not know for certain that the author is in fact a longtermist, but their consistent anthropomorphization of a pile of statistical formulae certainly suggests they wouldn't feel out of place among a crowd of such people.
In contrast, the people who prefer the term "ethics" in their work are grounded in real and present issues. They concern themselves with reasonable regulation. They worry about the current-day environmental impacts of training large models. In short, they are concerned with actual issues, rather than the alleged potential for a statistical model to "develop" sentience or exhibit properties of "emergent" intelligence (subjects from the annals of the science-fiction writing of last century).
I hope the author can clarify their choice of phrasing in their work, though I worry they have chosen their words carefully already. Readers should exercise caution in taking the claims of a soothsayer without a sufficient quantity of salt.
> LLMs do not "reason"; they do not "learn" or "develop" anything of their own volition.
If this claim of yours was sustainable, your grievance with the writing style would make a lot of sense. But GPT-4 can clearly reason about novel problems which were not in its training data. I think you should consider reading some of the many citations in the document which describe examples of that happening.
Really, if it always produces output that conforms to "expectations of a reasoning actor", isn't that defacto a "reasoning actor"? What's the use of making the distinction?
Because of all the ways in which it's obviously not a reasoning actor, starting from the design of the system, and all the ways in which it is literally engineered, by design, to output text that sounds like a human wrote it, based on consuming the Internet.
> In all ways except .. its output? Which is the only thing that matters?
Who decided that this was the only thing that matters?
The thing that concerns me isn't that it answers questions in a way that resembles a human. What concerns me is that some people think it thinks like a human, which it does not, and writing about it in an anthropomorphic way exacerbates this problem.
The issue at hand is not the observable output of the system, but the way in which the system works — which we have knowledge of. It's not magic, it's not emergent, it's not sentient, and it's certainly not intelligent. It's a (very fancy and impressive!) predictive text system. Imbuing it with extra properties is disingenuous.
My rationality and agency is not a question of you interpreting the words and letters that appear in a textbox on a screen. The same is not true of a large language model. You do not have a rigorous theory of rationality and agency with which to interrogate a large language model.
LLMs clearly develop internal representations, this is an empirical fact. For example, one paper cited is https://arxiv.org/abs/2210.13382, which trained an LLM on Othello games, where an 8 layer model is fed one of 60 tokens (there are 60 possible moves in Othello, 8*8 minus 4 initially occupied squares). You can probe this model at layer 5, and it computes Othello board state from moves, and you can be sure this board state is used, because if you patch activation so that board state is modified, the model generates moves for this modified board state.
> LLMs clearly develop internal representations, this is an empirical fact.
you've offered an anecdote. Some LLMs (more generally, this type of neural network model) will generate configurations that can be understood as a representation; others will not. The fact that the authors were able to find an apparent model in a heavily rule-based system is not incredibly surprising but offers little clue about whether this is "clearly" true for similar architectures that represent very different (real) world systems.
From your citation itself:
> More broadly, it would be interesting to study how our results generalize to models trained on
natural language. One stepping stone might be to look at language models whose training data has
included game transcripts. Will we see similar representation of board state?
Okay, I think we are in agreement. Let's say current LLM architectures are clearly capable of developing internal representations, not just learning surface statistics, and it can execute algorithmic computation as complex as computing Othello board states, and it can completely generalize out of distribution thanks to such algorithmic computation. (One experiment was to completely eliminate any Othello games starting with move C5 from the training data. The model had no trouble generating only legal moves for C5 games.)
I agree whether LLMs trained on natural language do develop world model is an interesting and open empirical question. They clearly can, but we don't know whether they do. I do consider the evidence pretty suggestive that they do. But this is clearly just an empirical question, not a question that can be answered by arguments as useless as "they are statistical models", because statistical models can and do learn world model.
I actually consider your considering this result not incredibly surprising pretty surprising.
The othello example explicitly was trained on the rules of the game - it isn't suprising that you can find a representation of game state.
LLMs are trained on the "rules" of human written language, which are sufficiently distinct from anything in the actual world that find world modelling would, indeed, be a surprise to me.
> I hope the author can clarify their choice of phrasing in their work
From TFA's conclusions section:
"Open debates over whether we describe LLMs as understanding language, and whether to describe their actions using agency-related words like know or try, are largely separate from the questions that I discuss here (Bender & Koller, 2020; Michael, 2020; Potts, 2020). We can evaluate whether systems are effective or ineffective, reliable or unreliable, interpretable or uninterpretable, and improving quickly or slowly, regardless of whether they are underlyingly humanlike in the sense that these words evoke."
>LLMs do not "reason"; they do not "learn" or "develop" anything of their own volition. They are (advanced) statistical models.
As a non-expert in the field I was hesitant at the time to disagree with the legions of experts who last year denounced Blake Lemoine and his claims. I know enough to know, though, of the AI effect <https://en.wikipedia.org/wiki/AI_effect>, a longstanding tradition/bad habit of advances being dismissed by those in the field itself as "not real AI". Anyone, expert or not, in 1950, 1960, or even 1970 who was told that before the turn of the century a computer would defeat the world chess champion would conclude that said feat must have come as part of a breakthrough in AGI. Same if told that by 2015 many people would have in their homes, and carry around in their pockets, devices that can respond to spoken queries on a variety of topics.
To put another way, I was hesitant to be as self-assuredly certain about how to define consciousness, intelligence, and sentience—and what it takes for them to emerge—as the experts who denounced Lemoine. The recent GPT breakthroughs have made me more so.
I didn't say it's "not real AI", but of course whether I meant that or not comes down to a definition of "AI".
In academia (where I am, though my specialty is a step removed), "AI" is exactly the region of research concerned with statistical problem-solving. Machine learning is sometimes synonymous, though I haven't been able to determine whether this is consistent among all self-described AI researchers or just some of them.
Systems like GPTs are not capable of "thought", full stop. There is no ambiguity. If this is hard for you to accept, I'm sorry, but it is a fact. They have no agency.
I saw someone sketch a good explanation. They were responding to a claim that Chat GPT could "pass the Harvard admissions process". While it is true that the system can access sufficient data to respond accurately to questions on an exam, and it could be prompted to generate an essay, it cannot of its own volition choose to submit an application. It doesn't even "know" that Harvard is a real place to which it can apply, nor does it "understand" how it could go about "learning" how to find out about these things it does not know. It simply isn't capable of these things. (And even if it were that wouldn't necessarily be evidence of intelligence, but for now it's a sufficient distinguisher.)
> Wat
Can you explain what about my last point was "wat"-worthy? Do you disagree that it is morally problematic to pretend a thing is sentient when it is in fact not?
We may just live in a world where there’s room for more than one viewpoint and people are able to research different aspects of the same thing without getting into debates over dogma.
> We may just live in a world where there’s room for more than one viewpoint
AI safety is not a "viewpoint", it is a marketing strategy. The goal of the people leading that charge is to drum up FUD to increase investment in their useless snake oil companies. I feel no remorse for not being inclusive of their "perspective".
At what point are we allowed to use the "reason" or "learn" for AI then? Can we use these terms for non-human animals? If so, which ones?
I think LLMs are at the point where adopting an "intentional stance" (using Daniel Dennett's term) is reasonable.
> At what point are we allowed to use the "reason" or "learn" for AI then?
Maybe when the technology is physically capable of thinking for itself, rather than just being statistically likely to respond in a way that appears like the way a human would respond. Which may never happen, for what it's worth.
> I think LLMs are at the point where adopting an "intentional stance" (using Daniel Dennett's term) is reasonable.
I hadn't heard of this term so I looked into it, and wow, what a biased Wikipedia article it has. It's written with a completely partial stance. I've no idea whether to take it seriously, to be honest.
That said, I disagree strongly. LLMs do not "intend" anything, and pretending they do is (in my view) highly problematic. They're just (sophisticated!) prediction engines; nothing more.
While llm don't learn, they have now enough token input space and handle context well enough to manage a cyclic conversation.
While not learning, they can still run inferences on what they are fed as context within the token space. And because they are becoming so good at getting context out of structured data, they can use the context provided as a layer on top of their knowledge.
So you can at a very good approximation upload knowledge into them, and such knowledge can represent how to use tools to fetch data they don't know about - i.e. what langchain agents are doing.
Now of course that is not 'llm' some, is llm + code for informaron discovery, but that code is entirely supported and enabled by the existence of good enough llm
People routinely make up their own vague and ill defined meanings of understanding and reasoning to disqualify LLMs. This is necessary because LLMs obviously reason and understand by any evaluation that can be carried out.
Seriously just watch. He's not actually going to be able to coherently define his "reasoning" in a way that can be tested.
> Seriously just watch. He's not actually going to be able to coherently define his "reasoning" in a way that can be tested.
Google gives the following definition of the verb "reason":
> think, understand, and form judgments by a process of logic.
LLMs do not think, they do not understand, and they do not form judgments. They do not come to their own conclusions. They do not have the physical capability. They are statistical models, nothing more.
> LLMs obviously reason and understand by any evaluation that can be carried out.
>LLMs do not think, they do not understand, and they do not form judgments. They do not come to their own conclusions. They do not have the physical capability. They are statistical models, nothing more.
"LLMs don't reason because they don't understand" is not the bastion of genius you think it is. It's a circular argument that relies on whatever bespoke interpretations you have cooked up.
They don't form judgement or conclusions? Sure looks like they do. So what's the difference ?
What is GPT-4 doing then when it correctly looks like it is reasoning and what's the difference between that and "real" understanding or reasoning.
Such a huge difference I should be able to test for it. Don't understand how you can tell me what I'm seeing isn't real reasoning but fail to provide a way to empirically determine the difference.
New bar for people claiming LLMs can't reason: invent a specific, testable problem, representable in text, that many humans can solve and LLMs can't, and tell us what it is.
That's not stuff an LLM can't do. It's just presented in a way that makes it difficult to do so.
First the vision problems will require the equivalent of an artificial visual cortex, something we are seriously lacking in artificial intelligence at the moment. Image to text won't cut it here.
The vision problems will require something much more than an image to text objective task. It will require the equivalent of an artificial visual cortex. We don't have that yet.
… and then perform a careful search of books and the whole internet to be sure what you think is novel hasn’t been thoroughly debated somewhere on stackexchange.
If it's a stochastic parrot, then merely randomizing proper nouns and filler text should be enough to prevent its abstraction ability.
If you're saying that we can't use a problem if any analog of that problem has ever been described, you seem to be arguing more strongly that it is a general intelligence than I am.
I’m saying that people in my circle have been asking what they think are novel questions and getting interesting answers, only to find out that very similar content exists on websites we know are in the training set.
That’s not intelligence that’s computers having better memory than humans. Useful, certainly, but hardly skynet.
I don't think you're being clear about whether the questions were novel. If you discover your question was uncreative, surely e.g. some details, wording, facts, names, or numbers inside the question can be changed to defeat a model that is answering it from memory?
If you're saying that it is not possible to change the details enough to avoid the model being able to answer that type of question, I think you are admitting that the model has learned a generalized ability to answer questions of that class, and is not actually using its memory to answer at all.
I don't care about whether it learned that generalized ability from seeing examples of the question and answer, which it then deduced an algorithm for and generalized -- that's how most people learn most things.
The asker thought they were. They were not. The internet is big and human memories are not.
As an aside, I’m really starting to hate these threads on here, people are constantly reading words that aren’t there in search of gotcha-it’s-skynet. It’s not. It’s just pattern matching and randomness with a giant amount of information encoded.
> As an aside, I’m really starting to hate these threads on here.
I'm not sure what to say, other than that if you'd like to have less frustrating conversations, you could do better than showing up with hearsay where someone asked a question they thought was unique, but it wasn't, and it can't be modified to be unique and then asked again, and you aren't willing to tell us what it was, and possibly don't know yourself.
It is not possible to have a serious conversation about your claim, and that's not because it is being intentionally misunderstood.
> skynet
You're the only person mentioning skynet. The conversation is about a ridiculous claim made up-thread that GPT-4 cannot reason or understand anything, which is disprovable within a few minutes of using it thoughtfully.
from the beginning of time people have been overestimating the complexity of things like the human brain and attributing it to magical things (like a creator) far beyond our comprehension but what seems to be happening now is that some people are underestimating it.
This is the problem with non-operational definitions, because now we need to know how you define "think" and "understand" and "form judgments", to move on.
Instead, could you operationally define "reason" in a way that a human is, say, 90 % likely to pass the test and GPT is 10 % likely to do?
Yes, François Chollet released ARC(Abstraction and Reasoning Corpus) benchmark for this in 2019, and the benchmark can be scored automatically. Humans solve 100% of tests and GPTs solve 0% of tests and GPTs made exactly zero progress from 2019 to 2022.
Another issue is the vision side. The vast majority of multimodal models are working on essentially an image to text objective task. That won't cut it here. We need the equivalent of an artificial visual cortex. We don't have that yet
It is. Although I'd be careful with the conclusions.
The problems are presented in a way that make it difficult to solve.
The vision problems will require the equivalent of an artificial visual cortex, something we are seriously lacking in artificial intelligence at the moment. Image to text won't cut it here.
For the text there could be tokenizer issues. LLMs don't really have any problem with abstract analogical reasoning
https://arxiv.org/abs/2212.09196
When you hear someone speak, are you thinking their thoughts?
We are talking about a model of language and our expectations for what the words would look like if a person were using language to reason with. People are confusing one thing (their own interpretation of what they are reading) with another (a statistical model that is being driven to align with certain specific human expectations).
I’ve been imagining “reasoning” as meaning “referring to an abstraction”. The idea being if you wanted to write a script to tell you how far an arrow of a given weight will fly, you’ll probably use some physics simulation (or even just a coefficient to multiply the input weight by) to abstract the arrow in flight. Then you can answer questions around that abstraction.
The alternative is having 1000 different results for different kinds of arrows, and averaging out the results for the ones similar to the input arrow.
An LLM is working on text tokens, it’s trying to give the most statistically common next token based on everything that’s been fed into it. Does that statistical model abstract the objects and concepts it talks about? Eh? I don’t know
This is one of those comments that begs for actually using LLMs. Yes, an advanced LLM like GPT-4 will absolutely show evidence of an abstracted world model, correctly answering questions about where person X thinks object Y is, or what relation two people have to each other, or playing a text adventure game while clearly following an internal map it has created for itself.
So I get that the evidence is that you can do insanely complex things with consistency, like playing a text adventure. You pick up a thing and it stays in your inventory etc, ergo, it’s modelled a character with an inventory in a text adventure.
vvv
But, coming at this as *respectfully and curiously* as I can here:
^^^
I guess I’m a little more skeptical about attributing magic to something that I know is only working from words as a source.
Like if it’s seen NOUN really close to VERB a lot, it can “assume” (read: bias output) that NOUN will VERB in the context it sees it in, that’s an abstraction! But a weak one right? Language is pretty inaccurate. The fact that NLP was a whole booming research field a few years ago tells me that just parsing language consistently is a difficult problem.
So, I know I can’t give you any convincing argument there since we’re already mentioning how people are giving untestable definitions of “reasoning” and “abstraction” in this thread. And im no better in that respect here, BUT I guess my best inkling of what feels “off” is a lack of precision, inherent to the medium in which it operates which makes it feel like a true abstraction is impossible in an LLM.
> something that I know is only working from words as a source.
Your intuition about the architecture might be causing the distrust. Yes, the ends of the model are words in and words out (although GPT-4 is now multi-modal and can accept images as input too), but that doesn't mean the calculations in the middle are dealing in words. The "language" in "large language model" is misleading. It's a large generalized model that imports and exports language.
That is totally possible, the explosion of stuff around this space has kind of blindsided me, so I don’t have much intuition for precisely how these models are built.
It all still feels rather “attribute magic to complexity” to say that there’s some human element skimmed off the patterns of our language, hidden in the coefficients somewhere.
I’m also not a magician, so really who am I to say :)
Does it really provide the statistically most common next token? I don’t think that’s even true. And we’ve seen what looks like emergent behavior as parameters increase. I suspect it is actually creating abstractions at a deeper level than standard DNNs.
I’d be curious to know! I get skeptical when it starts moving into the “emergent properties of language” but language was built by fleshy neurons, it could be that some inherent pattern emerges in the language we speak that reflects the inner workings of our brain, which by trying to emulate makes a similar brain on the other end.
However, that feels a bit magical to me at the moment. Still interesting to be along for the ride humanity is going on!
> 6. Human performance on a task isn’t an upper bound on LLM performance
Is that true? Because LLMs are trained on the work of human: a LLM that learned all of the content of the articles would at best yield output equal to that of the human writing.
The reasons the author gives seem unfounded to me:
> First, they are trained on far more data than any human sees
If the human writer has access to google (everyone in modern society) this point is moot.
> In addition, they are often given additional training using reinforcement learning before being deployed.
If this is human in the loop RL, then the upper bound would still be the human training it. If it isn't, refer to #1.
I don't agree with you. Human knowledge itself is not bound only to the previous knowledge before it, so why wouldn't AI be any different? This is more of where AGI will shine, but I still think LLMs can produce novel innovation.
There's also a difference between having access to and being able to 'comprehend' all previous works in a given genre for generating music, that a human simply won't have the ability to process and recall when creating their own music and the AI could surpass. Humans simply having access to information is different than using information.
> If the human writer has access to google (everyone in modern society) this point is moot.
But a single human can only read so much information in a finite amount of time (their life). Sure, the same thing applies to LLMs, but the theoretical limits are orders of magnitude higher.
And while this probably isn't what you were talking about, it's also pretty clear that any time-limited task favours LLMs. There's no human on earth that could write a non-trivial short story in 10 seconds, regardless of quality level.
> If the human writer has access to google (everyone in modern society) this point is moot.
Does not follow at all. There’s a vast difference between being able to search for and internalize a tiny, tiny sample of all the available information, and literally having sampled and synthetized all of it.
> If it isn't, refer to #1.
Meaning what? LLMs use adversarial learning, the same thing that allowed AlphaGo to reach superhuman levels in Go.
9. With millions of people copying, pasting, and running code written by LLMs (often students who have no idea what the code actually does), it will be impossible to keep an intelligence that surpasses our own 'inside the box'.
I thought that perhaps the program just recognized that the only value that could be used to calculate any period was that frequency, and perhaps the model didn't really know anything about the Earth motion through its orbit affecting the average temperature, so I asked:
"Can any of the values be used to estimate the Moon's orbital period?"
and it replied:
"No, the code and the data used in this program are not suitable for estimating the Moon's orbital period. The temperature data being analyzed is influenced by the Earth's rotation around the Sun and other factors like weather patterns, but it doesn't directly correlate with the Moon's orbital period."