The calendar adjustments are because the planet's constant orbital period isn't a whole number of days.

The leap seconds were an attempt to have wall clock time map to the planet's rotational angle consistently despite the problem that the planet's spin varies unpredictably.

Yes the "leap hour" is a legal fiction of course. In reality in the event anybody cares about this in the distant future they will make the kind of "drastic" changes you've probably experienced twice a year for your whole life and barely noticed... More likely because the drift is so incredibly slow they won't change anything.