This is one of the most ridiculously awesome findings this year, that changed my mental model of how foundation models work.
It sounds crazy, but it's way cheaper and faster for multimodal models to read text on images than the equivalent text represented as tokens. Text tokens take a lot more bytes to represent internally than you would think. Give it images with rasterized text, and there is no OCR-like intermediate step that turns images into text tokens. It just goes straight into the multimodal context where it's all just linea algebra underneath. The cost is more lossiness than if you gave it the same tokens as text. Which is why you don't just render text prompts and files to read as images.
But compaction is inherently lossy, a summary 1-10% of the original length is going to lose things, by definition. So you keep the things you really need to get exactly right in text token summary, but put a lot more of what you would have just discarded into a PNG. Keeping the past context as text in an image is way less lossy than keeping a text summary of that context.
They peeked inside the layers and found the same neurons were usually activating in the middle layers when fed either text tokens or images of that same text. It just takes the model more time and kv cache to do it via text tokens.
This is one of the most ridiculously awesome findings this year, that changed my mental model of how foundation models work.
It sounds crazy, but it's way cheaper and faster for multimodal models to read text on images than the equivalent text represented as tokens. Text tokens take a lot more bytes to represent internally than you would think. Give it images with rasterized text, and there is no OCR-like intermediate step that turns images into text tokens. It just goes straight into the multimodal context where it's all just linea algebra underneath. The cost is more lossiness than if you gave it the same tokens as text. Which is why you don't just render text prompts and files to read as images.
But compaction is inherently lossy, a summary 1-10% of the original length is going to lose things, by definition. So you keep the things you really need to get exactly right in text token summary, but put a lot more of what you would have just discarded into a PNG. Keeping the past context as text in an image is way less lossy than keeping a text summary of that context.
They peeked inside the layers and found the same neurons were usually activating in the middle layers when fed either text tokens or images of that same text. It just takes the model more time and kv cache to do it via text tokens.
Deepseek was the first to really jump on this: https://github.com/deepseek-ai/DeepSeek-OCR/blob/main/DeepSe...
[dead]