M5 is almost 2x the single core performance of the M1 Max. You would notice that things are faster.

But everything is fast enough, so moving to the M5 seems like a downgrade, as it is heavier

Well when macOS feels heavier now than before so is that 2x really a 2x?