I suspect it’s partially an average / regression to mean.
If you built a model only based on long form journalism and The New Yorker voice, including both the template structure of the article and the grammar and syntax, the model would replicate the patterns it is built upon. You could do the same for high vocabulary rap.
When you stuff all of English or all of human thought into it with somewhat equal weight, including very technical writing the sum is a lack of style and elegance and personality.
The second problem is that writing often doesn’t happen linearly. You write bookends and fill in, edit and revise. You sort of know where you’re going and can then make decisions how to get there. Language models may benefit from being redesigned to perform passes. Select a template of some sort, fill in the mad lib, then run a detail pass, then run a persuasion / emotion pass, then run a style pass. Current model writing very much sounds like I would expect autocomplete to.
It’s heavily RL’d and it’s gotten worse because they’re using LLM judges, that’s why it’s gotten worse. The last good writing model was GPT 4.5.
I suspect they’re leaning heavily on some kind of auto verifier, and that’s why the writing sucks, because good writing is so subjective.
And they’re probably nerfing what it can get from training as they’ve lost lawsuits for scraping copyrighted work, so they’ll purposefully avoid anything that references a specific writers style, etc.
Isn't this easy for frontier 'labs' to fix? If the user requests lots of writing and prose, then use an auxiliary model which has been trained on 'award-winning' writing instead of all the crap on the internet?
Award-winning writing has all kinds of different styles. The average of them will probably get you some boring median soup once again, though maybe a bit less bland than the LLMs trained on oceans of LinkedIn and Reddit slop.
Even if it doesn't, the effect of so much writing getting pushed out the door through default LLM settings is going to make it feel same-y and bland in short order.
Oh boy, more IP theft and copyright infringement!
The "regression to the mean" or "it's just the training data" hypothesis doesn't work. Pretrained models do not have these weird writing styles and sound much more human. It's only after instruction tuning -- SFT, DPO, etc. -- that LLMs develop weird writing styles.
That is, the original training on massive text corpora seems to produce an average of human writing, but that average looks pretty average. It doesn't feel like LLM slop. But the instruction tuning on specific tasks (answer this prompt, summarize this text, do this math problem), which turns text predictors into chatbots, seems to create strange style features as a side effect. And because that tuning step depends on company-specific training data (rather than just a crawl of the entire Internet), each company's LLM has a different style.
It's not that difficult to measure this. Linguists have had measures of style variation for a long time, though NLP and AI researchers seem to have forgotten. We've generated parallel human and AI corpora to measure it, and can show specifically which writing features change: https://arxiv.org/abs/2410.16107
Currently we're working on tweaking the instruction tuning process to see if we can pin down which steps and what types of tasks tend to produce the weird style features.