Am I dumb or does this chart make no sense? Or why does the line only go up even with compaction? Or maybe "overall trajectory size" is hiding some meaning I don't understand?
Am I dumb or does this chart make no sense? Or why does the line only go up even with compaction? Or maybe "overall trajectory size" is hiding some meaning I don't understand?
The chart makes sense and is describing the cumulative cost of a trajectory. Cache tokens are created when a trajectory’s prefix is used more than once. A larger pre-compaction context window means that a greater number of cache tokens are used per turn, and a larger number of turns are completed before compaction runs. So you get a cumulative cost that grows quadratically until the compaction event.
The blue line (200K context) is lower than the red line (300K context). Indicating that it's cheaper to run blue rather than red over longer cycles.
The Y axis is total cost in USD. For it to go down would mean OpenAI refunding you money.