Think of the entirety of the context (the full thread of conversation, all tool call output, etc.) as one message that's been submitted to the LLM all at once just so it can generate the next token (which is just the next word or even syllable.) Once that token is generated it is appended to the context, and the entire context is once again used to generate the next token. Keep doing that until the entire response is generated. The larger the context, the more stuff the model has to pay attention to as it generates each next token.
To use an analogy: imagine your friend is the author of an unfinished book. They die with 19 chapters written, and on their death bed ask you to write the 20th chapter. Assuming you're up to the task, you can only do this well if you take the time to absorb the entirety of what's been written so far.
This is how LLMs work. Context caching is an optimization on top of this, but it has its limits.
I guess it does increase their cost, or rather your share of their hardware depreciation.
AI serving cost is apparently mostly hardware depreciation rather than operating cost (electricity etc), and if your large context request is occupying VRAM for some fraction of a second then you are paying for the depreciation that occurs in that time!
e. H100 costs $20-40K to buy, with a lifetime of maybe 3 years, and will only consume maybe $2K in electricity if run 24x7 for those 3 years.
Think of the entirety of the context (the full thread of conversation, all tool call output, etc.) as one message that's been submitted to the LLM all at once just so it can generate the next token (which is just the next word or even syllable.) Once that token is generated it is appended to the context, and the entire context is once again used to generate the next token. Keep doing that until the entire response is generated. The larger the context, the more stuff the model has to pay attention to as it generates each next token.
To use an analogy: imagine your friend is the author of an unfinished book. They die with 19 chapters written, and on their death bed ask you to write the 20th chapter. Assuming you're up to the task, you can only do this well if you take the time to absorb the entirety of what's been written so far.
This is how LLMs work. Context caching is an optimization on top of this, but it has its limits.
I guess it does increase their cost, or rather your share of their hardware depreciation.
AI serving cost is apparently mostly hardware depreciation rather than operating cost (electricity etc), and if your large context request is occupying VRAM for some fraction of a second then you are paying for the depreciation that occurs in that time!
e. H100 costs $20-40K to buy, with a lifetime of maybe 3 years, and will only consume maybe $2K in electricity if run 24x7 for those 3 years.