I feel like, at higher latitudes, it should matter less?

Whether sunrise is at 3am or 4am, you still basically need to have a way to sleep with unreasonable amounts of sunlight (eg 18-20 hour days).

At lower latitudes, summer time makes more sense because the difference between a 5am and 6am sunrise goes from "too early" to "manageable". Same goes for 9am being too late compared to 8am in the winter for example.

At higher latitudes, you have to accommodate no matter what.

At lower latitudes, 1h makes a difference.