You need to be extremely careful about taking photos of children without explicit permission from the parents.

You need to know the law. That is being careful, but it doesn't mean it's always illegal to photograph children in public.

For sure, but it didn't help me because I don't have the fortitude to stand my ground. I'm very non-confrontational.

(Yikes-- I feel my pulse in my neck and chest just writing about this.)

I likely need to see a therapist about it. Wow.

At least on the street and in sports my experience is people using purpose-built cameras get harassed.

People using cell phones as cameras get a pass (at least in sports).

It's common that they don't even let you into the venue if you have a interchangable lens camera. I wouldn't even try going into a pro game with a camera if I didn't have a credential.

At my Uni I usually go right in without any trouble, the only case I got hassled was a woman's hockey game and that time I kept repeating "I've never had trouble getting into a game before" (true) until they gave up and let me in. (Which doesn't leave me inclined to try again, but I'd already bought a ticket and didn't want to back and stash my gear in my office) I hear in hockey they are really worried about wildcat video streams.

Some of the sports at Cornell are exceptionally laid back. We are one of a few schools that plays sprint football which is 100% the same as regular football except players have to weigh less than 178 lbs [1], I know the head coach, I know people in the parent's association, they leave the gate unlocked and i go right down to the sidelines.

[1] https://en.wikipedia.org/wiki/Sprint_football