It seems like if latency is having such a big effect that it's changing the winners, maybe your tests are awful and shouldn't be so latency dependent?
I mean, I get it: how fast a model responds is relevant. But a test that changes second by second is far less relevant than a test that tells you how smart the model is, and accounts for latency in some way that isn't constantly changing the result.
Latency isn't changing the results for the coding index or arena ELO, but neither of those take latency or throughput into account, so we added those to our leaderboard as score components.
Latency and throughput matter a ton as a user, so I think it's actually totally defensible for a leaderboard to bounce around a lot as these numbers change. The best model to use changes a lot based on these!