If everyone was an "investor" it means they aren't blowing all their spare cash on goods and services. Demand drops and you need fewer people to work to provide said goods and services. It kinda balances out.