>Also, we've realized the scientific reality that traveling faster than light is likely impossible, and the vast distances to other habitable planets would mean tens of thousands of years of travel even with the most efficient technology.

This makes for more epic story telling though. I've seen sci-fi stories span eons due to the vast distances limited by speed. Even at light speed it still takes a huge amount of time.

It rarely has to do with the possibility of the genre actually happening in reality. A story is good because it is riveting and the backdrop (whether sci-fi or fantasy) is just icing on the cake. Look at Harry potter and practically every fantasy story out there.... none of those stories have any chance of happening in our current reality... yet they are popular.

I think the answer for as to why there's less interest in sci-fi is pretty mundane. It's like genetic drift, but this one is cultural. For no reason at all our interests just shift because of luck. There happened to be more good stories written in the fantasy genre and that's the direction everything shifted.

Tbf though the MCU was more sci fi than anything and that every genre out of the water in every medium all the way up until endgame. Say what you want about it but the popularity of that universe dwarfed everything else in the last decade... which goes to show that it's likely mostly just drift.