The majority are really not that hard to categorise.
In the UK at least, fantasy and sci fi occupy the same shelving. Takes me ages pulling books out of the shelf, and immediately rejecting because they are fantasy.
The majority of the books are fantasy, not sci fi. Fantasy seems to have a much bigger audience in the UK anyway.
I'm in the UK. We must frequent different places, because I've never had that problem.
Well, it's a reasonably big place. It would be surprising if we did frequent the same places!