With thousands of token per second output it would be an enormous waste of resources. Such chips are clearly made to process thousands of conversations simultaneously. Not necessarily in parallel. All LLM workflows are turn based right now, there are often seconds between turns until tool calls finish or users type the next message.

If the LLM response only takes a few milliseconds, the chip can process hundreds of other requests until the first conversation becomes active again.