Strongly disagree with most of the consensus here to start with Player of Games and that Consider Phlebas is boring. I found Player of Games to be the much weaker book, with lots of heavy handed social critique and a generally quite stock plot.

It's true that Consider Phlebas is set outside The Culture. I think that's for the best as The Culture is pretty alien and the Phlebas protagonist is more relatable. Banks does a great job of building the world such that when you get to the end of the book, you're like "I get it" in terms of understanding the Culture. Plus, Phlebas has a number of wonderfully evocative set pieces that are super cool. You can see how influential the series was on later sci-fi, especially stuff like Halo.