The output distribution is altered - it starts responding "yes" 20% of the time - and then, conditional on that is more or less steered by the "concept" vector?
The output distribution is altered - it starts responding "yes" 20% of the time - and then, conditional on that is more or less steered by the "concept" vector?
You're asking it if it can feel the presence of an unusual thought. If it works, it's obviously not going to say the exact same thing it would have said without the question. That's not what is meant by 'alteration'.
It doesn't matter if it's 'altered' if the alteration doesn't point to the concept in question. It doesn't start spitting out content that will allow you to deduce the concept from the output alone. That's all that matters.
They ask a yes/no question and inject data into the state. It goes yes (20%). The prompt does not reveal the concept as of yet, of course. The injected activations, in addition to the prompt, steer the rest of the response. SOMETIMES it SOUNDED LIKE introspection. Other times it sounded like physical sensory experience, which is only more clearly errant since the thing has no senses.
I think this technique is going to be valuable for controlling the output distribution, but I don't find their "introspection" framing helpful to understanding.