Only a tiny, tiny fraction of the parameters are encoding information that's specific to a particular programming language. Even if you could remove those without degrading performance, it would have a negligible effect on the model size.
Imagine our LLM as this one-dimensional matrix (instead of unfathomably many, many more dimensions):
0.728201017
-1.9293839303
2.93020202929
5.82290101728
-0.2844920917
…
And then, all of these values are multiplied by some scaling factor, essentially vectors pointing in some direction (in our case, only on a one-dimensional line instead of in some higher-dimensional space).
Now imagine it’s possible to “remove” the portion of the weights’ directions that’s responsible for some undesired property like history knowledge, or another programming language:
0.428201017
-1.6293839303
2.63020202929
5.52290101728
-0.5844920917
…
(For demonstration and faster-to-type-out purposes, I’ve changed each vector by deducting a constant amount of 0.3, which doesn’t make much sense. In reality, it’d modify probably all digits in some direction.)
Now we can see: the number of rows has stayed the same, but the value has changed. This means the required storage (or computational) space remains the same.
Please do correct me if this simplified too much or I had a mistake in my understanding.
This is oversimplified. The proposed question is whether a model with fewer parameters could achieve performance on one language similar to that of a larger model that’s been trained more broadly, which isn’t straightforward to do.
Oh, I didn’t interpret the above question as asking in that direction; but yeah, that’s of course something I didn’t attempt to answer with my comment.
Although I’d be intrigued in the answer to that small-narrow vs. large-broad model question, too!
The exception being the token embeddings and lm head (which scale with the number of tokens the model knows and presumably you need a smaller number in the tokenizer for only English and python). But those are a pretty small % of the total model weights on most LLM sizes
For a set parameter size, if you benchmark an LLM on a single language, you'd get better results by training it on multiple languages rather than just the one. Scaling quality training data almost always increases performance as the neural network picks up general rules and an understanding of code that are language agnostic.
Only a tiny, tiny fraction of the parameters are encoding information that's specific to a particular programming language. Even if you could remove those without degrading performance, it would have a negligible effect on the model size.
Not who you responded to, but I’ll give it a try.
Imagine our LLM as this one-dimensional matrix (instead of unfathomably many, many more dimensions):
0.728201017
-1.9293839303
2.93020202929
5.82290101728
-0.2844920917
…
And then, all of these values are multiplied by some scaling factor, essentially vectors pointing in some direction (in our case, only on a one-dimensional line instead of in some higher-dimensional space).
Now imagine it’s possible to “remove” the portion of the weights’ directions that’s responsible for some undesired property like history knowledge, or another programming language:
0.428201017
-1.6293839303
2.63020202929
5.52290101728
-0.5844920917
…
(For demonstration and faster-to-type-out purposes, I’ve changed each vector by deducting a constant amount of 0.3, which doesn’t make much sense. In reality, it’d modify probably all digits in some direction.)
Now we can see: the number of rows has stayed the same, but the value has changed. This means the required storage (or computational) space remains the same.
Please do correct me if this simplified too much or I had a mistake in my understanding.
This is oversimplified. The proposed question is whether a model with fewer parameters could achieve performance on one language similar to that of a larger model that’s been trained more broadly, which isn’t straightforward to do.
Oh, I didn’t interpret the above question as asking in that direction; but yeah, that’s of course something I didn’t attempt to answer with my comment.
Although I’d be intrigued in the answer to that small-narrow vs. large-broad model question, too!
The exception being the token embeddings and lm head (which scale with the number of tokens the model knows and presumably you need a smaller number in the tokenizer for only English and python). But those are a pretty small % of the total model weights on most LLM sizes
For a set parameter size, if you benchmark an LLM on a single language, you'd get better results by training it on multiple languages rather than just the one. Scaling quality training data almost always increases performance as the neural network picks up general rules and an understanding of code that are language agnostic.