A few months ago I had to work around a problem like this and the best out there was WhisperX. Not sure about transformer.js support.

Link to the repo - https://github.com/m-bain/whisperX