Quite the opposite. The earth's rotation can vary by quite a few seconds each year. But over hundreds of years, the variations mostly cancel out, which is why I think trying to add or remove leap seconds are a bad idea. The only people who care about such things are space agencies, and they can apply whatever offset they want, it doesn't need to affect everyone else.
Arguably, the only real-world impact this drift has on normal people is GPS, but GPS already transmits an offset from its own clock so that receivers can correct for that. The GPS clock is different both from UTC and TIA.
> the variations mostly cancel out
I don't think we know this do we? we haven't been measuring accurately enough for long enough to be confident that it does in fact cancel out. In fact for the period of time where we've been applying leap seconds they were happening with significant frequency and always in the same direction. It's only been very recently that there's been the realistic suggestion it might drift in the other direction.
If anything people have been suggesting that if we do get rid of leap seconds that we can just wait until the offset is enormous instead (say an hour) because it would take a very long time to happen. But even still, that does actually affect everyday people (because people would surely notice when solar noon is an hour later/earlier). Although pragmatically the obvious solution there is to change timezone instead.