For those not trying, this allows Deepseek to understand a picture (instead of just extracting text from it), and it can describe what's in the picture, but this is not an image generation system, so you can't ask it to modify an image.
Personally, I'm a bit surprised the DS chat app still doesn't offer its own text to speech and speech to text features (I know DS doesn't have any ASR model for example, but there are quite a few in the open).
DeepSeek interpreting screenshots and images I send it at fractions of what I pay Claude and ChatGPT, for me, is of far higher priority than supporting dictation. There are workarounds for dictation but not image processing.
You can do that with smaller models at home. Gemma-4-E4B will run on a 12gb GPU, and supports audio, image, video input
12GB GPU is a lot
just use one of the various cheap gemini models
Indeed, Gemini really is incredible at image analysis. Yesterday I pointed it at some sloppy handwritten notes and asked it to add up the numbers in the right column, and it did it no problem. I've also used it to find out what TV show or actor is on screen, and various other things. It's quite impressive.
Gemini pretty clearly has the best underlying model, and the worst RL and post-training of the lot.
gemini models are also fantastic at understanding non spoken sounds
Or you could just use a CNN...
CNNs are not SoTA anymore when it comes to large models, and also are not used to provide interpretations of images as text, but rather to classify, do semantic segmentation, etc.
CNNs are fine when trained with a good recipe. There are very few good studies comparing them with proper hyperparam search and all the training tricks applied consistently. Transformers are good but ViT vs CNN is not some settled issue. Transformers are more hyped and more popular with the tech enthusiasts who just read forums and news, but if you need stuff done, CNNs are still great.
I agree, but since we're talking about imagine understanding with text output, clearly a CNN is unsuitable. My previous comment was overly reductive and CNNs can still be SoTA depending on your performance metrics. I spent the earlier part of my career training CNNs, and they are very pleasant to work with.
Can you say more about that? I haven't kept up.
CNNs excel in vision tasks where you have limited compute, limited memory, limited data, and want something that works super well and quick. People usually don't hook CNNs up to a transformer to get language understanding either, you have to train bespoke CNNs for specific tasks
ViTs excel where you're unbounded in compute + data and also want text understanding or have a conversation about an image
Transformers are superior
Which?
Can you explain what the benefits are of actually "talking" with the bot instead of typing and reading?
As someone who would rather send a slack message to a coworker rather than actually walking over and talk to them, the idea of having to talk with my laptop is not appealing at all, haha.
If you spend your life sitting in a chair, that's fine. I tend to get all kinds of ideas, questions, and research needs while I'm walking around. Typing a paragraph or two or context takes too much time and is very risky. Especially when driving. But also just walking, cooking, cleaning, etc. Sometimes it's just not practical - winter, carrying stuff... I mostly feel privileged if I can just sit at a computer and type my question and have the time to read the answer.
It’s crucial to use for driving/walking.
One problem has been ChatGpt/Claude apps don’t really do this well. They use weak and/or non-reasoning models for voice interaction and the UX is not optimized for hands free.
I wrote an iOS chatbot app mainly for this purpose for myself and family/friends. Allows starting/sending voice prompts with the action button so I never have to look at the screen. Supports any model at any reasoning level so conversations are not dumbed down. Added a video transcription tool so any model can “read” YouTube/Tiktok videos and chat about them. Great to discuss lectures on tech topics.
It takes slightly longer to use a reasoning model for voice interaction use but I prefer the intelligence. The latency can be minimized a few ways, bidirectional streaming helps. It’s TTS agnostic, I’ve got a few selectable providers and the output can be prompt styled “use a chill tone that’s not too eager”.
I mean, even applied voice 'models' suck for this.
For some godawful reason, Apple Maps voice directions assume that you also understand what it omits. So if it says "turn right in 500 meters" "250 meters" and then you stop at an intersection after 150 meters and it says "turn right", it expects you to understand that it doesn't mean the immediate right at the intersection, but the next one [because you still haven't driven the full 250m]. It is nuts and I have no clue how that has ever gotten past testing.
What it should do is say nothing until I have to turn, or say "turn right in 100 meters" "turn right".
This is one thing Waze I think seems to do better than the competition. And they have a ton of different voices.
They also clearly show which voices can do street names (which is hugely helpful). For some reason the Australian and British accented voices feel more polite than the Americans
What are the use cases of an LLM while walking or driving, that also require high reasoning?
With a sufficiently sophisticated harness you can actually do quite a lot by just talking to your AI. I have regularly dictated to build things on my phone while walking to lunch for example.
Most of the problem is that for voice chat, you usually get no reasoning at all and no tool use at all to research or ground assumptions.
For example for voice ChatGPT still uses a quantized gpt40 non-reasoning model that hallucinates pretty frequently. It also doesn’t do much automatic search for updated information and fact checking.
I usually don’t find I need high, usually DeepSeek v4 with medium reasoning is sufficient.
However if it’s important chat like brainstorming on complex topics I sometimes bump it up.
OpenAI has a new voice api that supports adjustable reasoning, but ChatGpt is not using it currently.
Gemini 3.1 flash live is a native audio to audio model with reasoning. But it's still not a SOTA level model
I am someone that prefers a slack message to a coworker than talking to them and I use AI.
My current flow is: Google Eloquent to capture 127WPM (my typing is best case is 65wpm). This lets me get the thoughts out without thinking too much about structure or flow, the same way I would brain-dump type it.
Next I use AI to compress, summarize, and restructure to create a clear coherent message for my peer to read (which is way faster for them).
When communicating with AI, its the same thing, except I skip the second step since AI does a good job at understanding my ramblings.
----
It drives me crazy that some cultures only send voice messages to each other. It drives me crazy they can't be respectful of my time and use STT+AI to convert their 90 second monologue to a few written sentences.
Slightly off-topic but: does it concern you that you're letting atrophy a very important skill for human communication (organising your thoughts and ideas, and then clearly communicating them to others)?
Tbh, I never have been a good writer. A college professor once told me I am a terrible writer. I've tried to get better (I read a lot, I write a lot, I've taken multiple college level writing course). I even started a blog (https://kcoleman.me).
I kinda view myself as a wheelchair user. I'm bad at walking so I use at wheelchair so I can at least have a semblance of decent communication. I don't think my ideas are not worth sharing, but I'm just bad at writing them in an engaging way.
The scarier thing for me is coding. I am good at coding. But I don't even read a single line of code any more.
This worries me tremendously. In fact, it is one of the major points of value that i deliver as an engineer. Organizing and iteration on thoughts is not trivial or easy, but it is very important!
As someone who's still learning English, this is one thing I'd never use AI for, at least not in the near future, simply because thinking and structuring my thoughts before typing is the same as it is before speaking and actually talking to other people can't be outsourced to AI.
But I imagine if I'd been a native speaker I wouldn't mind using AI like OC does since it's a convenience. Same way I use a calculator for two digit multiplications in real life but spent years learning to do it manually in school.
You're probably further into english than I am into vietnamese, but I really like using AI to help me improve my vocabulary and understanding of the language.
I avoid using AI as a direct translation tool, but its super useful for me to translate complex english ideas to vietnamese.
As a native English Speaker I can tell you that I would have some trouble talking out an email. I like the back and forth in my head of editing as I go. Text messaging may be fine but email is more difficult for me to just talk through.
I am loving the conversation here though of how people are using speech to talk to LLMs or not though, it is something that no one talks about much
I would find this behavior extremely aggravating from a co-worker. If you can’t be bothered to edit down your ramblings by hand, just don’t send me anything at all.
Maybe you missed my point?
I can either edit down my rambling by hand (costs about 10-30min depending on the length) or I can ask chatGPT for assistance, where I manually edit chatgpt's edits for factual correctness and tone.
--- STT
Like, besides the lease risk, I think 30 to 50% of the business is going to end the moment the owner stops showing up and the motorbikes are gone. Either, I think it was Roger mentioned or you mentioned the Google reviews all mentioned that people go to the bar because it attracts other motorcycle people. And, you know, we don't have an existing, like, network to come in and grow this. And so we might see a 30 to 50% decline in revenue within a month or two with nothing ready to, like, backfill that with. And if our main goal is to make a private club or event space, ideally, I'd like to have some, like, something substantial to, like, commit to that spot. Like, right now, we're three fucking dudes with, you know, a little bit of a vision, but not really. And, you know, we're buying what will be a failing bar unless we, you know, figure out how to run events or use that backspace.
polish
---- gpt5.5
Polished version:
Separate from the lease risk, I think there is a real chance that 30–50% of the business disappears once the current owner stops showing up and the motorbikes are gone.
Either Roger mentioned this, or we discussed it separately, but the Google reviews seem to reinforce the same point: a meaningful part of the bar’s appeal is that it attracts motorcycle people. We do not currently have an existing network or community that can step in and replace that traffic.
That means we could see a 30–50% revenue decline within the first month or two, with no clear plan in place to backfill it. If the main goal is to turn the space into a private club or event venue, I would feel much better if we had something substantial already committed to that location.
Right now, we are three guys with a loose vision, but not much concrete traction. Without a clearer plan for events, memberships, or activating the back space, we may effectively be buying a bar that starts failing the moment the current identity and customer base disappear.
> It drives me crazy they can't be respectful of my time and use STT+AI to convert their 90 second monologue to a few written sentences.
I have used Whisper to transcribe audio into text in the past. You could probably build a pipeline for that, whether running locally or in the cloud - and the run the transcription through the same summarization agent.
Just my two cents: I have coworkers who use AI to drive basically all their communication in Slack and I absolutely hate them with a deep passion. I actively avoid meetings, conversations, and exclude them from everything possible.
If you use AI to drive your communication with other humans, you suck.
Sending me your AI compressed ramblings = straight in the bin
What did you do prior to 2023?
I hardly type at all now. I use Handy (free) with Parakeet and use its post-LLM processing feature with a custom prompt tailored towards coding, so I can say things like "Have it go to slash remote dash control" and it'll output "/remote-control". Converts brackets, etc.
Everything is almost instant, it's insanely fast, and lets me work on multiple different agents/windows at the same time fast with cmux.
I use the same thing to talk to people on Slack, iMessage, etc now when I'm working from home instead of typing.
I also can help articulate my thoughts better when I'm thinking them literally out loud instead of just sitting silent and typing them on a computer for hours.
It's just something that you need to try and get used to because I also thought it was something I wouldn't like at first.
Can you share more information on the post-LLM processing and the prompt you use? I would like to try this out but don't see any post-LLM options in Handy.
edit: nevermind, found info on the docs about how to enable post processing. Would still be interested in your prompt though if you don't mind sharing!
You have to enable "Experimental Features" under "Advanced."
This is the prompt I use (it's probably overkill and can be condensed):
https://pastebin.com/raw/RUVAqLCU
What is Parakeet?
I believe this is the correct link. I use it too in Handy, for English and Spanish transcriptions: https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3
Maybe they meant narakeet?
https://www.narakeet.com/tools/
Parakeet is the name of a speech to text model from Nvidia. Roughly comparable to whisper from openAI.
It's the model doing the work inside the wrapper that an app provides.
Yep, here's the v2 and v3:
https://huggingface.co/nvidia/parakeet-tdt-0.6b-v2
https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3
It's almost instant on my new M5 Max w/ 36GB of memory, but I used both with Handy on my previous 2019 Intel Mac w/ 16GB memory and was completely surprised at just how fast it was for being on-device! Not instant, but only a couple seconds.
I’m using it on an M3 max 32gb, and I’m getting 60-70x realtime for recordings and crazy good accuracy. I can get an hour of audio transcribed in a minute. Similar results from Whisper, but half the speed.
Transcription this good used to cost A LOT, now it rounds down to free.
I type as fast as I talk so for majority of my LLM usage I don't need text to speech.
But I love the chatgpt voice interface e.g. on a long drive when I can use it to learn about random stuff (btw, turn advanced voice off for such usage).
Other part though is, hacker news vs regular population, majority of which would much much rather talk and listen than type and read.
I like to talk (stt) but I don't want tts to talk back to me I just want to read the response. voice synthesis is a waste for me personally.
I thought this way until I tried it, and the main difference is that when I'm managing tons of agents at once or just reviewing some plan / approving next steps, or need to give quick feedback/ask a simple followup, the voice interface makes me much faster and more likely to continue because it's lower friction (and in many cases that's good, though not all) and can be hands-free.
Actually, my thoughts on this matter changed so much that it inspired me to get much more into voice controls because I realized how this same problem was basically why some people sucked at remote work or weren't able to properly use tools like claude code, because it was essentially the same problem but worse (typing / messaging feeling too high-friction or raising the barrier for participation). I have a way to let Claude call me now to tell me stuff when I have a bunch of instances out doing stuff and then leave to go home.
I'm trying to get that better integrated in my devloop because I think it makes managing >4 agents simultaneously much more feasible and natural for some people (I used to play Starcraft a lot so I'm used to the multitasking, but it still takes sustained willpower to be constantly "driving" or monitoring things, or to field questions), especially ones who have never served as TLs or people managers before. IMO it's a big performance roadblock for a lot of developers to be treat directing multiple agents simultaneously as some kind of high-stakes/high-cost thing. The kind of developer who would not say anything in a team meeting unless prompted or who thinks everything is stupid by default (because they are afraid of making decisions / being wrong even if only briefly) is both very common and reluctant to work this way, but also really probably needs it to be as productive as more skilled developers.
I don't know about you, but I force myself to read the whole spaghetti thought process of any AI that's actually working on code, and make sure I understand what the hell it just said before I ask questions or give it a green light. Even or especially when whatever it said is full of fluffy stuff about having understood the problem space. That's usually where a well-placed question can bring the entire structure crashing down.
"You're right to push back" has become the gold standard phrase I'm looking for from these things to assure myself that I'm covering all the bases and understanding what it's building (not that that's enough, and not that it isn't still going to build some ungodly blob anyway).
I kinda like using voice to jot down my next questions or iterate on things, but there's a clear danger to it, which is that you may inadvertently be signing off on stuff you haven't thoroughly read. If there's one thing about LLM-written code, it's that the devil is in the details.
[flagged]
Faster, and that's it. If you don't need precision (like with prompting LLMs) the speed gain is massive (*for most people)
I've been using ChatGTP by voice for things like cooking and house repair stuff. It's quite convenient for situations in which your hands are busy.
Other week I fixed a a water valve. After planning the thing with ChatGTP I brought the new valve. Then I described what I was seeing as I swapped the old valve for the new one to make sure everything was right. Really cool experience!
When I was still using OpenAI, I used it among other things to translate from English to Spanish while talking to Spanish-speaking people in person.
I understand a bit Spanish but I don’t speak Spanish yet, and they don’t speak English.
I speak English to the AI and end with “translate to Spanish, translation only”, and then the AI says the thing I was saying in Spanish (not perfect but good enough, and also it has a slightly weird accent that might be it using English or English influenced text to speech even when speaking Spanish sentences?).
Sometimes it's faster than swyping on a phone, but mostly I use it to learn about stuff and hash out ideas while driving.
This may sound strange and even callous, but I think it's appealing to people who are used to having employees. It's not about speech being a better interface, it's that thinking hard enough to sit down and compose a prompt is too much work if you're used to just yelling at someone.
Pity the managers with no one left to boss around besides the machines coming for their own jobs.
I was asked just yesterday if I could wire up [redacted] so that [redacted profession] could have a realtime voice interface while in the middle of performing [redacted]. My basic answer was yes, but it would be a bit slower than you want if something is going wrong, and it would probably be unethical for a whole lot of reasons.
Accessibility.
What about accuracy?
I'd imagine it'd be a reasonable tradeoff for disabled people who can't use their hands.
Much faster and better flow. Don't knock it til you've tried it.
A lot of people are slow typists.
I can talk faster than I can type.
it's very confusing. maaaybe if the stt is good and fast enough, speaking may be faster? english speakers can probably hit 150-180 wpm but seems like a hassle
It's easier, faster, and more natural to talk than to type for the vast, vast majority of people.
This trivial fact of life is observed every day by e.g.:
- students taking notes and finding it necessary to only jot down key facts so that they can keep up,
- stenographers who require special training and equipment to keep up verbatim with live speech in the courtroom,
- annoying colleagues who insist on "hopping on a quick call" or arranging big, wasteful, and disruptive meetings instead of just writing down their problem / sending a message or email,
- friends who insist on sending short voice messages in DMs instead of typing, because it's more "personal" that way (which to be fair it is, but not to the extent proclaimed).
Also vision can be used for "compaction" https://blog.can.ac/2026/06/10/snapcompact/