this is cope (i mean, it happens a lot, but is not the cause of what is seem on the paper).

I noticed even with plenty of context, the model sometimes chose a path that "fools" the attention layers to bypass some rules. you can notice that the rules are not ignored, they are sidestepped. it's very predictable emergent behaviour after you see it happen. with more context, it actually gets worse to the point you will see some things obfuscated even.