This is true for a large subset of problems. LLMs are bad at basically all games, for instance. LLMs are even bad at chess, which has a huge amount of publicly available data. LLMs are really bad at games like backgammon, which has only 10s of millions of tokens of public data.