This can be extended even further - hunter-gatherers never retired either. However in both cases people did actually _retire_ from direct activity. Elders would often become advisors and community nurturers rather than actively milking cows, hunting deer and whatnot.

I think here what becomes apparent is that it's not loss of specific activity (work) that causes the decline but activity in general which is very much duuuh - obviously.