I'll admit I am not an expert in the field, but the fact that "chain-of-thought" optimisations function by getting the model to extend its own context window with more language to me hints that what we consider an "intelligent" response is ultimately contingent of the language processing.
In any case though, if language is just the input/output modality, where is the intelligence when language is not involved? Is the "intelligence" of ChatGPT/Claude/Gemini models dependent on the human-decision-curated linguistic dataset they have been trained upon, or is it prior to that? If a SOA LLM were to be trained on the same dataset as them but was not in any way put through RLHF for it to respond to human prompts, would it be intelligent? What would be the expression of that intelligence?
I also achieve better performance on cognitive tasks when I use language to first describe the problem I'm trying to solve. In fact, it usually helps quite a bit (see: rubber-duck debugging)
I'm not sure the word "intelligence" really fits what these models are doing. I do however think it's safe to say that they are performing cognition - even if it's 'simply' cognition over their provided context and even if it's entirely limited by their training set. We still have a machine that can perform automated cognition over a increasingly wide distribution of data.