> the variations mostly cancel out

I don't think we know this do we? we haven't been measuring accurately enough for long enough to be confident that it does in fact cancel out. In fact for the period of time where we've been applying leap seconds they were happening with significant frequency and always in the same direction. It's only been very recently that there's been the realistic suggestion it might drift in the other direction.

If anything people have been suggesting that if we do get rid of leap seconds that we can just wait until the offset is enormous instead (say an hour) because it would take a very long time to happen. But even still, that does actually affect everyday people (because people would surely notice when solar noon is an hour later/earlier). Although pragmatically the obvious solution there is to change timezone instead.