Cached.
they are the first part of the input and it contains no user dependent variables, so the model is in a known state that it can reuse across all users, it does not need to recompute all that inference
Cached.
they are the first part of the input and it contains no user dependent variables, so the model is in a known state that it can reuse across all users, it does not need to recompute all that inference
Unless they are using a linear architecture, the compute cost still scales O(n²) for n tokens, and nemory cost scales O(n).
>the compute cost still scales O(n²) for n tokens,
That is never the cost, it's a common misconception.
Cost scales linearly per tokens. Unless you are sending one token at a time and avoiding using the same machine or cache.
Just look at api charges, they are charged by token, not by token squared.
Which seems to contradict the usual consensus that purely linear architectures are not sufficiently capable and unsuited for frontier models.