Hi HN, I'm Ruud, co-founder of Lont. We turn company emails into personalized videos.
Video captions cost us manual work twice: fixing misheard words, then fixing layout, like lines ending on “the”.
Overtone uses multiple models to listen to the audio: gpt-transcribe, Azure STT, and Elevenlabs Scribe. When they return conflicting results, Overtone sends small clips to the models again, and then lets an LLM decide what was actually said.
The LLM is like a human doing a sanity check on the transcription model. Humans would easily recognize a word to be different based on context. The LLM basically does the same, using an optional context prompt + its world knowledge. Because of this the Word Error Rate (WER) on many test files is lower when using Overtone, compared to just using a model without it.
Sure, the cost is higher because we use up to 3 different STT models + an LLM, but it still removes mistakes, reducing the time needed to fix captions to a minimum, and reducing the chance you'll miss something. For us it's worth it. For ultra high volume use cases it may not be.
The rules-based cutter nicely cuts the transcript into captions that look good, depending on screen size: 16:9 is different from 9:16, and the Overtone Format we created supports any type really. Pauses become breaks, captions are grouped per speaker, and names/numbers are never spread out over multiple captions.
How to add these captions to your videos? A videojs plugin can do this automatically, so you don't burn in your captions. The plugin lets you cycle between styled captions, plain captions, and off. You can burn them into an MP4 file with overtone burn.
Quickly try it out just the cutter? Try:
npx overtone-captions recut yoursub.srt
We open-sourced this first version (MIT) because we believe many people can benefit from this, especially if your captions just need to be good. There's plenty of room to improve though, so we've made it easy to help contribute to add style packs, languages, and providers/models.
You can find a demo at: https://overtone.lont.ai/
Find the code and benchmarks here: https://github.com/lonthq/overtone-captions
This is a great idea. I recently had this exact problem when captioning a demo video in portrait mode. And to my surprise there was no easy way to automatically fit the captions on the screen. Will definitely this tool a try.
Awesome, please give us your feedback, the current version was purely built for ourselves so we probably missed things other people need.
Experimenting with STT myself, I understand the need for re-reviews :)
I feel that conveying that is hard to the end-user as the cost of STT is real yet they don't always appreciate that given the ease of access to dictation these days.
It's pretty simple right, a frontier LLM like Opus 5.5 has substantially better results than a cheap locally-run model.
Talking about that btw, gpt-6.1-terra scored substantially worse on fixing punctuation in our tests than gpt-6.1-sol. Didn't expect that really.
[flagged]
love the idea, how many languages can you support?
It currently only supports a handful of languages but there's no limit to what can be added really.
Only one language per video is tested currently, multi-language is a bit trickier but would probably still work, especially if it's just a few terms in another language. Haven't tested.
I have no idea what’s going on, but modern subtitles are suddenly TRASH. All kinds of misheard words, it really feels like they removed all human oversight, or reduced it so significantly it’s just impossible to actually read it all for whatever crew works on this.
I only glanced through capabilities, so apologies if I simply missed this, but I am begging you to ensure there is some kind of system for translating foreign text on screen - most ideally, in a slightly differentiated font.
So often someone is walking by a sign in some other language, and their comments on it are translated, but the sign is not! I just watched Sai on Netflix, and it changes to a bright red screen with white text near the end of most episodes. I think it’s introducing the credits but I have no idea. It was just 30 seconds of Japanese text.
Misheard words is something that Overtone fixes. But it can't fix all, in some cases audio mix and pronunciation are so bad that it's genuinely hard even for a human in their native language to understand. Music also messes up speech-to-text models.
Unfortunately translation of (on-screen) text wasn't built into Overtone. This is something both Netflix and Disney are very aware of, and they already create multiple visual versions of content. On both platforms you can trigger a different visual language version by changing your UI language. Disney movies are most known for this: https://www.youtube.com/watch?v=FxzL7UYyAeg
They use the IMF format to transport content with "overlays" for different languages. In those languages certain frames are swapped out for a translated version. So that an EXIT sign is seamlessly translated to a SALIDA sign in Spanish for example.
Simpler way to do this is forced narrative captions. They are basically a separate captions track to only translate stuff like the on-screen sign you mention. This is combined with regular captions. Netflix turns them on by default I believe, and on Apple TV+ it's called "auto" captions.
The core problem is that AI isn't yet able to do everything. There's a race to the bottom going on for translation companies and things won't get better until AI gets better probably. Companies like Netflix and Disney still spend good money to get high-quality captions, but smaller ones don't unfortunately. Best thing for you to do might be to message the provider so they at least know that people are unhappy.
I’ve tried looking for instances of specific movies or shows doing this sign translation thing but haven’t found anything other than some Disney stuff. Do you know if this is used anywhere in non-animated film? Or if anyone puts translations near the sign, styled to look similar, rather than replacing it?
Netflix does this as well, they did it for a Mexican show called Ctrl-Z: https://www.youtube.com/watch?v=amyCefA5PfM
And here's an example of the Warner Bros. movie Demolition Man, where they localized Pizza Hut product placement outside of the US: https://www.youtube.com/watch?v=mn-PD4VuG8g (even the logos on the door were paintbrushed away, there's more clips to be found on YouTube btw)
Both Netflix and Disney push development of the Interoperable Master Format (IMF). I haven't seen visual localization happen outside of those two.
The John Wick movies have "expressive" closed captions, which as far as I know they've done in both English and Spanish. Other languages get forced narrative subtitles, sometimes with and sometimes without the English burnt-in captions.
And about the future: GenAI could definitely help localize parts of the video. Wouldn't surprise me if it's been used for this already. Combine that with IMF and you've got a solution that can instantly be adopted in the high-end production world.
[flagged]
[flagged]
[flagged]