I don't have the same experience at all. The people who I know and have met which enjoy watching sports come at all levels of fitness. I am not that interested myself but quite many of the members of my running club enjoy watching sports and some of them can run 3 hour marathons.
And if you have ever met football ultras most of them are very fit.
Are you coming from UK or US perspective? Never heard the term Ultra.
What are football ultras?
Somewhere between dedicated fans of (often working class) European Football teams and paramilitary organisations. Green Street starring Elijah Wood is a film accessible to overseas audiences detailing the cultural phenomenon n the UK.
https://en.wikipedia.org/wiki/Ultras
In Europe however, Italy in particular, organised crime is embedded in the structure - e.g. Gennaro Di Tommaso aka "The C0rpse", the leader of Napoli's ultras whose approval is needed before a match can take place
https://www.reuters.com/world/europe/curva-nostra-mob-infilt...
Extremely fanatic soccer "fans", aka hooligans. Basically just an excuse to be antisocial.
the mob that pretends to be the core fans.