So-called "caveman" thinking attempts to address this.
The important part of "actually wait, I really need to XYZ" is just "XYZ".
The model can attend to just "do XYZ" and produce almost the same vector modifications as full verbose "reasoning".
So-called "caveman" thinking attempts to address this.
The important part of "actually wait, I really need to XYZ" is just "XYZ".
The model can attend to just "do XYZ" and produce almost the same vector modifications as full verbose "reasoning".
I don't think that's true. If a context contains a statement followed by something opposing that statement, that will confuse the model. So "actually wait, I really need to" is there to signal that the previous thinking may be flawed and that what follows is a new attempt.
It's good to remember that LLMs have no more state then what they can derive from the context up til any point. So if that context is hard to interpret, that will reduce effectiveness.
As long as the model itself understands that when it says "foo bar" in its own reasoning trace it means "pursue theory foo; no, this didn't lead anywhere, let's backtrack and pursue theory bar", where's the loss in clarity?