The racist accusation has been overused so much it's not just having diminishing returns, it's having no effect. If you can't argue someone's point but just throw labels at them, you are going to lose because the labels lose value the more broadly you apply them.
And that will allow the real racists to walk right through your gates because you can no longer identify them when everyone you don't like is a racist.
This is the real racist, walking through the gate.
The label pretty obviously fits in this case though? No one is calling DHH racist as some sort of slur; they're just describing his system of beliefs.