in the Culture universe, you don't have to stay in the Culture - you can get defanged and slum it out on some backwater hell hole like the rest of us.

Right. Just because the universe is post-scarcity, doesn't mean you can't build items, and entire production chains. Just because you can live to near immortality, doesn't mean you have to do things to prevent natural aging.

One guy became a member of the alien race the Affront, so named because they they have culturally and socially ingrained traits for exploitation, sadism and brutality.