Also people turning up in a town one day with no one to vouch for them were assumed to be up to no good as it could be assumed that you'd do just that if you were escaping your previous reputation. You could start with less than nothing by default, and may never shake it, and that's before race or religion.
> "If you weren’t born and raised here, you’re an outsider even though you’ve lived here for thirty-five years. That’s just kind of typical in small communities." https://dokumen.pub/small-town-america-finding-community-sha...
In small town New England, to this day, the litmus test is whether you graduated from the local high school, meaning you're probably related to other people in town. If you moved here at the age of 19 you're SOL.