That's not true for everyone. Some extraordinary musicians with perfect pitch also reported that it starts drifting for them as they age (I remember Oscar Peterson and Jacob Collier telling this)