It's not about directly regurgitating the training set, but about creating associations that allow it to indirectly regurgitate, and more critically - mix, its training set. It's easier to understand this in a different domain. Imagine we want to make a letter parsing algorithm.
The first step is to simplify the input and create a consistent format. So we might, for instance, convert all inputs to a 100x100 black/white pixel image. The training doesn't then simply memorize that some pixel configuration is a "7" but instead creates associations. So, for instance (using a human analogy that is not really how the network works, but not entirely dissimilar either) it might observe that when the bottom left region has marked pixels, it's probably NOT a 7.
Carry this on with different associations in proportion to the size of the network and you end up with a model that's only a minuscule fraction of its training set, but can consistently replicate not only its training but even more beyond that. Now change those 10,000 input pixels to words instead of pixels, with the target output being a word instead of a letter, and you get ChatGPT. It's the whole magic of neural networks, and really neat. But I think also paired with not really justifiable hyperbole.
Intelligence isn't especially well defined, so you end up lost in a semantic jungle there if you don't really define what you're talking about. In this context, I think most of us are implicitly talking about intelligence such as we would associate with humans. And what is that? 50,000 years ago we knew basically nothing, and even language itself may not have been developed. Now? We've discovered the secrets of the atom, put a man on the moon, and more. We seem to have clearly gone well beyond any reasonable interpretation of what our "training set" at the time was, and at an unbelievably rapid pace if one considers how extremely inefficient our march forward (and sometimes backward) is.
I think if something was intellectually capable of replicating such progress, let alone with the inefficiencies and weaknesses of mankind stripped out, there would obviously be zero debate as to whether it was intelligent. Just think about the implications. Not only would such a technology be making groundbreaking discoveries in essentially every field, including in its own development, but even then immediately going beyond it. Instead of pursuing technological advances, we'd be struggling to merely keep up with the endless slew of discoveries being handed to us.
I agree word play is always an issue. Keep in mind we start over every generation. We keep training on new data.
I've also thought about human development and what I find interesting is that an anatomical modern human is not automatically capable of forming modern civilization (see isolated tribes still living and general hunter gatherer history in general). There is so, so much that depends on emergent factors like some sort of global or societal intelligence. I'm thinking AI has the same characteristics. It's not about the individuals, it's about "society".
> 50,000 years ago we knew basically nothing, and even language itself may not have been developed
I think language is substantially older than that, but it is a hot area of debate.
The first step is to simplify the input and create a consistent format. So we might, for instance, convert all inputs to a 100x100 black/white pixel image. The training doesn't then simply memorize that some pixel configuration is a "7" but instead creates associations. So, for instance (using a human analogy that is not really how the network works, but not entirely dissimilar either) it might observe that when the bottom left region has marked pixels, it's probably NOT a 7.
Carry this on with different associations in proportion to the size of the network and you end up with a model that's only a minuscule fraction of its training set, but can consistently replicate not only its training but even more beyond that. Now change those 10,000 input pixels to words instead of pixels, with the target output being a word instead of a letter, and you get ChatGPT. It's the whole magic of neural networks, and really neat. But I think also paired with not really justifiable hyperbole.