The popularity of a genre is deeply tied to the historical context of what's going on in the real world. In the 1950s and 1960s you obviously had the Space Race, which itself was a product of the Cold War. Some commments view this as optimism. I disagree. I view it as an expression of patriotism.

I view this patriotism as a crutch in uncertain times. Another example of this is the rise of the superhero genre in the 21st century. Marven was skirting with bankruptcy in the 1990s. A perpetual Spiderman license was sold for a fraction of what it would today. And I think it's no coincidence that the superhero genre ascended post-9/11.

I used to read a lot of fantasy but many years ago I pretty much switched almost entirely to sci-fi. For me the reason was because sci-fi asked questions. I mean there's also space opera and it scratches the same itch as fantasy (IMNHO) but my personal interest in sci-fi is more in the "what if" category.

That's a very broad category that goes all the way from, say, the Iain M. Banks Culture series, which really poses the question of what an ultra-high tech post-scarcity civilization looks like to something like The Handmaid's Tale (yes, that's sci-fi). We've also had some superb sci-fi on the screen in recent years like Severance, Silo and The Expanse.

For me, fantasy is a far more limited genre. Like I'm trying to read Brandon Sanderson recently and while he's a good writer, it's just not hooking me yet in the same way that, say, Revelation Space did. I think the last fantasy series to really hook me was A Song of Ice and Fire. The first three books are some of the finest books ever written.

But as for the decline of starships, I think that the readership has evolved too. More vintage sci-fi simply projected the era of exploration and colonization onto space and it's become pretty clear how unrealistic that is because of the vast distances involved. You have to remember that popular media in the 1960s expected Moon bases and such in the not-too-distant future. Star Trek was really the last gasp of this and, interestingly, Star Trek is fundamentally socialist, which is noteworthy given that it originated in the Cold War.

Star Wars on the other hand was a tale of the resistance to imperialism. George Lucas has said he modeled the resistance on the North Vietnamese. Cyberpunk came about in the 1980s (eg Neuromancer, Blade Runner) that had nothing to do with starships but they really reflected a societal pessimism. Cyberpunk is inherently xenophobic (ie because of fears of the Japanese).

I wonder if the popularity of fantasy is fundamentally escapist and an expression of helplessness. Think about it: fantasy usually revolves around the outsized impact of individual actions, of a hero or heroine.