Prior to 1970 or so there wasn't an objective medical test used to tell men from women in sports and you had the problem of men who would participate in the Olympics as a women to win medals. And I think the frame was that, not "I am someone who lives as a woman"
My wife on the other hand thinks that gender-variant people are frequently alienated from their bodies and less inclined to participate in sports and other physical or nature-oriented things. I do photography for a running club in a very queer-friendly town which has me looking at the list of participants. We let people register as NB and have medals for the top 3 but frequently out of a few hundred entries we get 0 NBs even though running in a 5k on the flats would be an easy way to get a medal if you can pass for NB. On the other hand I've witnessed people set world records at more serious events and I can see people concerned about a record being set that natural women could never beat.
My strongly held view that seems to be profoundly unpopular is that I don't want the state settling the issue but rather it is a question for sports leagues to decide. Different leagues have different goals and different sports are different: I think transgender people should have a place where they can benefit from participating in sports and I think people have real concerns. We have a running gag about a male friend of our family who was one of the first players on an amateur women's hockey team because they were struggling to get a quorum -- he's not a big guy and not a great hockey player but he played a role in helping woman enjoy sports participation.