> maybe due to JS itself?

Nail on the head; a lot of JS overhead is due to its dynamic nature. asm.js disallowed some of this dynamic behaviour (like changing the shape of objects, iirc), meaning they could skip a lot of these checks.