Tons of buildings aren't residential. And there's still lots of single family homes as you get out into Queens, Staten Island, up further in the Bronx and further out in Brooklyn sometimes too.

I wonder how many people are living in buildings with 100+ people as I am (estimate a couple thousand in my building from back of napkin math I think).

New York City's fairly vast when you take into account Brooklyn, Queens, Staten Island and the Bronx. Manhattan's less than a tenth of the land area.