The current hardware used is self-hosting mini-server grade, and certainly not on the latest silicon process. "Slow" is expected.
It is not the ISA, but the implementations and those horrible SDKs which needs to be adjusted for RISC-V (actually any new ISA).
RISC-V needs extremely performant implementations, that on the best silicon process, until then RISC-V _will be_ "slow".
Not to mention, RISC-V is 'standard ISA': assembly writted software is more than appropriate in many cases.