> Under the hood, the models still work the same way (after training), and the analogy still applies, no?

Under the hood we have billions of parameters that defy any simple analogies.

Operations of a network are shaped by human data. But the structure of the network is not like the human brain. So, we have something that is human-like in some ways, but deviates from humans in ways, which are unlikely to be like anything we can observe in humans (and use as a basis for analogy).