> Each stack would get dirtied to the largest depth and that memory never gets reclaimed even if you only hit that deep of stack once

Right, but the issue under discussion was that "right sizing" stacks is hard. You always need to have a stack sized to at least the largest depth.

Pages that get swapped out to disk and never accessed again aren't that expensive. Disk is cheap.