Speech is in quite a weird spot when it comes to locally running it. Some of the best STT and TTS models run at faster-than-realtime speed on decent consumer hardware with quite small models (so also no huge RAM requirements), but the tooling and library ecosystem is much more fragmented, making it a lot harder to build actual working products around it. There is also a much more noticeable coverage / quality gap when it comes to languages other than English.
As a speech-to-speech model Dessa is of course in a different ballpark, but given the intelligence it exhibits, it seems to be working with not that many parameters. The developers also seem to be admitting that to some degree ("in fairness, LLMs also have to learn C++ and whatever the hell Angular is"). I'm wondering what the utility of this would be. Given that it lacks both broad world knowledge and tool calling integration in any way, that seems kind of like a dead-end.
Probably Dessa started those demo conversations with some context, so maybe it could receive documents and speak about them. If those were random conversations started with no context, the people at Block probably recorded a lot of uninteresting chat and cherry picked the few good ones.
What prevents them from adding tool integration?
How would those tool calls work? Would it say what it wants and another model/system act on that and 'read aloud' the document to it?
I guess that's somewhat how we do it, so maybe it would work.
It's a bit strange to call it a dead end when this was trained on just a single NVIDIA GB200 NVL72. LLMs couldn't do much back then, either.