I’ve tested at least 20 STT models in a benchmark I’ve set up with German, Italian and English voices from meetings in my company.
The voices contained very industry specific words, the languages changed from one sentence to another, sometimes words in a language were mentioned while a discussion was in another.
The only local model that satisfies me is Voxtral Mini 3b, the only paid API that is slightly better is eleven labs. Yes, Voxtral might not reach the best score in the benchmarks, but to me, it just solves a problem. It might not be the best in terms of speed... but that’s not a problem for me.
Happy to test this new model from Google but I’m not sure I’d go with that instead of something that can run so easily in my machine.
What i want is a model that outputs its predictions and their scores along with the text it choose. So I could flag something it's getting consistently wrong, like mis-predicting a technical term or name, or acronym, and say replace it with my correction, and have the ui be able to smartly replace that in the whole text so far and future parts. even better would be the ability to feed this back into the model for future runs.
It's interesting that the results can be so different depending on the person, the use case and even the microphone used. I use dictation a lot, so I try to stay up to date with the latest models as much as I can. So far, for my needs, nothing could beat Whisper Large v3. I keep hearing that Nvidia Parakeet models are better, but they just don't work as well for me, even though they are unquestionably faster.
Things change a lot if you need to speak to the model in multiple languages. There are very few models out there that can automatically detect the language spoken and produce correct output.
I tried the larger Voxtral models, but they didn't work for me at all. When I spoke to them in Polish, they produced output in Russian or Ukrainian.
For now, I settled on creating my own plugin for TypeWhisper, which runs Whisper Large on the GPU and does it much faster than pretty much anything else out there. But I'm still hoping that something better will come along, as Whisper Large is quite old at this point.
Agree about Voxtral being the only model — local or cloud — that handles multilingual conversation really well. I‘m not sure what they do differently, but mixed-language sentences and industry terms don‘t seem to faze it where other model begin to struggle.
Do you use it on a desktop? Mac by any chance? What's your setup?
I've been looking to find a simple and fast dictation app for English but almost everything I've tried (from Handy to many apps, eg some with Whisper in their names, after the model I assume) just don't work well. Apple's offering is worse than those though. I even tried with local enhancement models.
I am biased as a developer on the project but you should give Epilude [1] a go if you want local dictation on a Mac.
We’ve fine-tuned open-weight models to make them better (in our benchmarks) at cleaning up and formatting what you say so you don’t have to edit what you dictate.
https://epilude.com
I used MacWhisper for a long time and then recently switched to the free TypeWhisper. Both work very well on a Mac.
Out of curiosity, what was wrong with Handy? I use it and it works fine.
You should give VoiceInk[1] a go. It works great and has decent latency.
https://github.com/Beingpax/VoiceInk
This problem is legitimately hard and needs high cognitive abilities. Even the biggest generalist models struggle with memes and lingo salad that sound immediately intuitive for an out-of-the-loop human, and I'm talking about text comprehension. Modern models are optimized for decision making and are worse in that than old ones optimized for creative writing, but those also struggled.
You should probably not expect a small STT model like Voxtral or Parakeet to do any better, unless it's laser focused on that area in particular and sucks in everything else.
have you tried groq whisper-turbo? works well for me for multiple different languages
I use Parakeet 3. How does that fare in your testing?
I love it but it misses the business specific words when in different language. Sometimes it stretches them out to fit unrelated words in the language of the rest of the conversation. I miss its speed though.
Any possibility to share some of the eval audio?
Of course not, but it’s easily replicable just by mixing different languages conversations together, adding a word here and there of some very specific German jargon.
Try the recent MOSS one? It’s very good