I agree sci-fi is an outlier on this, but I also think all stories compete on setting to some extent. Fantasy most obviously (Tolkien, JK Rowling). But also for example the Jazz Age setting of The Great Gatsby contributed a lot to the novel's popularity and was a bit fictionalized, hard boiled detective writers like Hammett or Chandler wrote about a crime-filled world that was fictionalized for appeal, historical romances about lords and ladies are super fictionalized and so on. Writers try to put appeal into everything, that's why they're writers.
Larry Niven isn't referring to merely an "unusual" setting in his quote (which I've never managed to find referenced online, unfortunately), but to the way in science fiction you are creating the setting from scratch. Gatsby is set in the Jazz Age, and you can pick up some aspects of it from that, but it is still in the stock set of settings the author expected you to have some ideas about, so it doesn't explain how cars work or how doors open. And by that, I don't mean the sort of "explain" at an engineering level, but things like "how combadges work" in Star Trek, i.e., when they work, when they don't, what can be sent on them, what failures they are prone to, etc. Even something as fantastic as Tolkien is still generally set in a particular milieu and he is adding very skillful and numerous brush strokes to a genre that existed already.
You've read many stories set in all the settings you mentioned. You have never read a story in which the fundamental shape of space-time is two time dimensions and two space dimensions before, unless you have also read Dichronauts. This is the supplementary material to the novel, which is mostly not in the novel and is not the story itself, just the background: https://gregegan.net/DICHRONAUTS/01/World.html You don't need that provided for something set in the Jazz Age, or a fantasy story explicitly based on myths that had been floating around for centuries, or a historical fantasy. Someone could write some equivalent, but you don't need it; it's already loaded into your head. That's the point.