Insults cannot have greater harm than just about any physical injury. False accusations already have a legal recourse, as they're defamation.

Have you had a broken leg? When you're young it's an alright thing to deal with, when you're older it can be life altering (it might hurt forever or alter your gait). However most spontaneous violence doesn't result in broken legs but rather hits to the head which can very quickly end up at CTEs or other brain trauma.

In an equal society, you really don't want actual violence to be on the same spectrum as speech of any kind. One problem is that ~50% of the population has a massive natural advantage in the realm of actual violence. Would a husband beating his wife because of "violent words" she inflicted on him be ok under the "speech is violence" rubric?