I'm very interested in Diffusion text models. The concept of taking noise and adding words starting randomly all over the response, and filling in the noise from there on breaks my brain.
I'm sure I have a fundamental misunderstanding of the technology, though.
DiffusionGemma goes one step further even, and does this denoising over multiple "canvases" which lets it do reasoning and separate out a "final reply" canvas, looks something like this: https://gist.github.com/embedding-shapes/f4cb46bad704b6d0168...
Diffusion text models for me is the more interesting type of LLMs for local usage, as it really makes good use of single GPUs for single responses, rather than auto-regressive ones, and is a lot faster! Probably the fastest model I've been able to run so far, ending up doing ~670 tok/s (depending on the type of text) on a Pro 6000
Denoising is probably the weakest part of this model. There is recent research from Kaiming He showing that predicting the noise is actually not the best strategy for image generation since noise space is so large.
Simply predicting the surface of the data you are trying to generate is far more representationally efficient, and perhaps in the next few months we'll see a version of that for LLMs
Diffusion language models work with a discrete output space, unlike image models that repeatedly refine a continuous output, so they don't do the noise-prediction thing anyway.
Ok, you are correct. These models don't train noise predictors at all unlike first gen image diffusion
How does that break your brain? It's how basically every human writes and iterates on text..?
Because my brain thinks through text in a forward motion. Pausing at the end of each word and searching for the next.
My entire brain runs on sentences and words since I have no inner eye or whatever. So my thinking and writing both work kind of forward only.
I wouldn’t have thought that was too unique. But maybe it is?
Normally people have feelings about things before they are able to put them into words, I would imagine if you were asked a question like "what city would you most like to visit" then unless you've already thought about it a lot, then you would have to do substantial non-verbal thinking before you can come up with an answer, and once you have the answer you may respond "my favorite city is X" and you decided what X would be before you started the sentence.
Here's the best way I can explain how I "think" because it is mostly passive, but sometimes, I am actively thinking.
If I'm asked a hard question, like what city I'd most like to visit, I kind of have to look up cities and cross reference them with ones I've noted that I want to visit, or have visited and liked enough to go back, then actually read the list, and figure out which one, but each time I read a city, I might have a, this would be great in summer or winter or they have a cool festival in June. Which leads to additional question I have to ask. Which I think about roughly the same speed I speak, maybe a word or two faster.
My wife is the complete opposite. She can put together a sentence in an instant in "parallel" so when she says it, she though it already, but not as a sentence.
Human brains all work so crazily weird, and we are only just now started to realize all the nuances.
Does a human brain start with pure noise and then move that noise subtly towards the target?
From what I've noticed at least for myself, there are like at least two distinct parts that don't always cooperate. There's a draft part that's random thoughts that just appear and can be converted immediately to speech, but when writing it's actually another part that takes dictation from that one, albeit silently.
If I'm just typing blind I'll somehow end up writing random homophones down with completely correct spelling, like hear instead of here, it's bizarre. So that part is sort of structured autogenerated noise that is then consciously either appended or inserted into random spots in text. Maybe it's more of an LLM first pass then diffusion refinement.