> There is some talk of eliminating the leap second, which would over time have the Earth and sun diverge with regards to noon and such

“Over time” really glosses over how much time it would take. In 500 years there might be half as much divergence between solar noon and 12:00pm as we intentionally inflict on ourselves with DST, or that France and Spain inflicted on themselves in the 1940’s so they could share a time zone with Germany. By the time anyone will even notice we will probably change time systems for other reasons anyway. It’s not even remotely comparable to the Julian/Gregorian issue, which dealt with leap days. Each day has 86400 seconds.