Wouldn't say that racists were in the minority. Hitler starting several wars turned them off from fascism, in Britain as well, but regular old racism and white supremacy was strong.