Hacking meme aside, lots of computer scientists are overextending their domain expertise into an area that has been well studied by philosophy and biology. It isn't surprising the software is good and the philosophy looks like outsider art.
Here are some relevant pointers to connect this discussion to the existing philosophy on the subject: https://en.wikipedia.org/wiki/Emergence
https://en.wikipedia.org/wiki/Emergentism
Basically there's a lot of cases where some properties arise from sets of a thing, which would otherwise not be present in a single or few things.
One classic example is that a single molecule or drop of water does not express fluid mechanics.
And of course, a bit more basic would be materialism, but maybe you are one of the lucky hundreds:
Yes. There's nothing essentially new this latest round of AI has unturned that philosophers haven't turned over decades (or more ago). Nothing stopped philosophers supposing even a functionally perfect simulacrum of human intelligence, and getting technologically closer to it doesn't.
The real effect of the latest round of AI has been inducing software engineers to be pretend-philosophers as they're approaching this set of questions for the first time -- and are having a very hard time engaging given their enthusiasm for technology.
Armchair philosophy without empirical data is just stuck in a loop of endless thought experiments. LLMs basically nuked the Chinese Room argument out of existence.
John Searle treated understanding like some magical binary property you either have or you dont. LLMs proved that understanding isnt a static noun, it is an emergent phenomenon.