I bet you could get the same performance with smaller models, but as english-only, by removing training data in non-english (keep code, translate in-code strings mechanically prior)
I bet you could get the same performance with smaller models, but as english-only, by removing training data in non-english (keep code, translate in-code strings mechanically prior)
I doubt multi-lungualism is taking up that much space in the weights. Languages are systematic, and most of them share a lot of features and can be compressed into the same representation together.
The real problem is all the random facts about the world that are not easily compressible, and you want the LLM to know a lot of random facts.
AFAICT there are several things that affect the size of the model:
1. the number of transformer layers -- e.g. each layer adds the ability to provide more information to the residual stream [1], [2]; each layer is `tokens * features * active context`
2. the number of tokens -- the more tokens there are the more word fragments and Unicode characters you can represent
3. the number of features assigned for a token -- these are used to associate trained semantic information for the token; with the active context this can encode contextual information for that token (the idea being that the model can differentiate senses of a word like "bank" depending on its context, so "bank (river)" will have different weights than "bank (building for money)")
4. the active context -- this is the number of `token * features` to include when predicting the next token
One of the key features of LLMs and transformer architectures is that training across different languages helps the model learn the underlying concepts and associations, so training on English only would lose that.
A better approach may be something like a byte architecture or a model where the tokenization is trained with the model (similar to vision and audio) instead of as a separate step. Or maybe training smaller models longer to take advantage of double descent [3], [4].
[1] https://arxiv.org/abs/1512.03385 "Deep Residual Learning for Image Recognition"
[2] https://www.youtube.com/watch?v=QgH9sr7G13Q "The most cited paper of the century is a brilliant hack"
[3] https://arxiv.org/abs/2403.10459 "Understanding the Double Descent Phenomenon in Deep Learning"
[4] https://www.youtube.com/watch?v=z64a7USuGX0 "What the Books Get Wrong about AI [Double Descent]"