That's always going to be a trade off with anything like this so I guess it's better to think of it as an alternative to compaction.
Another use case that comes to mind is that sometimes I'll include some detail early in a conversation and I mean it as incidentals information but the AI fixates on it. If I could selectively edit that out rather than start a whole new conversation it would be worth the cache miss.
I have not noticed a measurable slowdown in practice so far, including canvases with around a hundred nodes. A request only includes the wired ancestors of the current node, not the entire canvas, so node count alone is not a good measure of prefill cost.
That said, your concern is valid for very long contexts. Editing an early ancestor may reduce prefix-cache reuse, while pruning a branch also makes the resulting prompt shorter. ThoughtDAG does not manage its own KV cache today, so this is something I need to benchmark properly rather than claim is solved.
Have you encountered this mainly with local models or hosted APIs?
That's always going to be a trade off with anything like this so I guess it's better to think of it as an alternative to compaction.
Another use case that comes to mind is that sometimes I'll include some detail early in a conversation and I mean it as incidentals information but the AI fixates on it. If I could selectively edit that out rather than start a whole new conversation it would be worth the cache miss.
I have not noticed a measurable slowdown in practice so far, including canvases with around a hundred nodes. A request only includes the wired ancestors of the current node, not the entire canvas, so node count alone is not a good measure of prefill cost. That said, your concern is valid for very long contexts. Editing an early ancestor may reduce prefix-cache reuse, while pruning a branch also makes the resulting prompt shorter. ThoughtDAG does not manage its own KV cache today, so this is something I need to benchmark properly rather than claim is solved. Have you encountered this mainly with local models or hosted APIs?
[dead]