> what an index describes vs what investors prefer

What makes you think S&P 500 did not become the most popular index (instead of full market ones) because of the rigid entry criteria and and rules for weights.

Amongst other things the weighting is not even based on the market cap.