In the author's own analogy of blacksmithing and metallurgy, I see an interesting parallel.

Humans worked metal for a long time and you can make better and better forges without knowing the metallurgy of why the result is better. If I make the fire hotter the metal comes out better, and I can get to work making forges that produce hotter and hotter fire.

LLMs could in this analogy be the forge. We can make them bigger and bigger and get better and better answers out, in the same way a pre-metallurgy human could make their forges hotter and hotter and get better and better metal out.

But the hottest forge doesn't mean you get metallurgy.