This matches with what Nicholas Carlini from Anthropic said a the [un]prompted conference - https://www.youtube.com/watch?v=1sd26pWhfmg. Very worth watching.