The S&P500 has increased 9.8% annually the last 100 years, roughly 6% annually adjusted for inflation. Yes, past performance is no guarantee for future, but historically a completely passive index placement of wealth into S&P500 would double the real (adjusted for inflation) wealth every 12 years. With absolutely no work.
Also, if you are wealthy enough you can just wait out any economic downturn. Hell, Im not even that wealthy and it would have to get really bad before I would be forced to sell in a down market.