Why not ignore gender labels and go by chromosomal configuration? There could be XY and XX [1] olympics. And then there should be X, XYY, XXX, XXXY, XXYY, and all the other possibilities [2].
There is more complexity than the binary in the expression of sex in humans.
[1] https://en.wikipedia.org/wiki/Sexual_differentiation_in_huma...
That is what they do. Male, female, man, woman, boy, girl are sex categories, not gender categories, that is they predate the very idea of gender as distinct from sex.
Sports categories never had anything to do with gender.
The other difference of sexual development are different sexes
All biological categories are fuzzy around the edges. Those fuzzy edges do not invalidate the category. The existence of small #'s of people with actual physical intersex conditions (not "I feel like <x>") in no way conflicts with humans being sexually dimorphic.
I agree with you in general, but I think it would be fair to let XY individuals with CAIS compete on the female side - their bodies do not respond to testosterone.
Really, there should be separate categories for people with more than the regular amount of arm hair. Also separate categories for short people, tall people, lazy people, people who wear glasses, people with blue trousers, and of course, for sketch artists and quantitative traders.