If you ever played 20Questions you know that you don't need 1000 dimensions for a billion concepts. These huge vectors can represent way more complex information than just a billion concepts.
In fact they can pack complete poems with or without typos and you can ask where in the poem the typo is, which is exactly what happens if you paste that into GPT: somewhere in an internal layer it will distinguish exactly that.
That's not the vector doing that though it is the model. The model is like a trillion dimensional vector.
With binary vectors, 20 dimensions will get you just over a million concepts. For a billion you’ll need 30 questions.