That might actually boost performance since attention pays attention to stuff that stands out. If I make a typo, the models often hyperfixate on it.