Some of that could be related to the ISA but I'm hoping that it's just the fact that the current implementations aren't mature enough.
The vast majority of the ecosystem seems to be focused on uCs until very recently. So it'll take time for the applications processors to be competitive.
The RISC-V ISA can be fast.
Tenstorrent Ascalon, expected later this year, is expected to be AMD Ryzen 5 speeds. Tenstorrent hopes to achieve Apple Silicon speeds in a few years.
The SpacemiT K3 is about half as fast as Ascalon and available in April. K3 is 3-4 times faster than the K1 (previous generation).
This should give you an idea about how fast RISC-V is improving.