One term of art that's emerged for this "feel" is "big model smell", first coined by @aidan_mclau. [0]

To my surprise I couldn't find any proper explainers of the term in a quick search, despite grokking it after seeing it in various contexts on Twitter, but Fable 5 offered a useful analogy: "A student who memorized worked solutions and one who understands the subject score the same on the test; you can only tell them apart by asking a question the test didn't. Real-world use is nothing but those questions, which is why a single AA number feels right and wrong at the same time."

In other words, big model smell is related to the underlying ability to "understand" when tasks are underspecified or out-of-distribution. This ability can be mimicked to parity by smaller, distilled models according to the density of the training data for particular tasks, but neural scaling laws still hold for generalized reasoning ability.

More recently with these smaller models, there's a separate but related "RL-fried" phenomenon, where they rely on CoT to "grind toward a checkable answer even in contexts (open dialogue, taste, ambiguity) where there is no checkable answer, and you get the tell: over-hedged, over-structured, relentlessly on-task, deaf to the subtext."

There are some other insights and caveats in the (short) conversation that I feel you may appreciate reading. [1]

[0] https://x.com/aidan_mclau/status/1807843014104211855 [1] https://claude.ai/share/d511a348-7c36-432f-a6d5-9deab2802615