Exactly, I 100% agree, and IMO toolchains should default to assuming fast misaligned load/store for RISC-V.
However, the spec has the explicit note:
> Even though mandated, misaligned loads and stores might execute extremely slowly. Standard software distributions should assume their existence only for correctness, not for performance.
Which was a mistake. As you said any instruction could be arbitrarily slow, and in other aspects where performance recommendations could actually be useful RVI usually says "we can't mandate implementation".