So the interesting thing is that this shrinking “linguistic diversity” is fundamental to how an LLM works.

The LLM is a big probabilistic statistical trick. It picks the next token based on certain words are simply “the best” because they are specific and well connected to other tokens. The is gives them a great overall cost function. (Basically a good score on “will it make sense in context” while also having specific meaning that makes it better than other options, unambiguous in common use and being a single token rather than several).

You can trim those tokens, but then you just get other tokens that are “the best” tokens (and you’re worse off because the output became less clear).

The cool thing is this seems to get worse the more powerful and accurate your model is, because it is picking technically / statistically perfect tokens, not tasteful ones.