It is insane that people think we need a growing population to make this perfect population pyramid, to make things work easily in monetary terms with taxes. It really does ignore so many of the other forces as you mention.

The picture looks radically different if you focus on real resources and allocation. In fact a growing population could make things very economically tenuous in real terms, depending on how a few key environmental factors play out over the coming centuries.