I don't think this is a limitation of the JVM. When I've used Clojure's STM implementation it's been perfectly serviceable (barring the composability issues I mentioned). Likewise when I've used the various STM libraries in Scala. Eta (basically a Haskell implementation on the JVM that unfortunately stalled in development) also had a fine STM implementation.

It's more of a combination of API and language decisions rather than the underlying JVM.