OMP changed the default compaction to images! Kinda nuts to read about. Saves the generation cost of the traditional compaction step and writes the context as tiny text to an image, if I was following correctly.
OMP changed the default compaction to images! Kinda nuts to read about. Saves the generation cost of the traditional compaction step and writes the context as tiny text to an image, if I was following correctly.
But what if the model you're using doesn't have image processing capabilities?
You can set different compaction strategy, currently "Summarize in place and keep the current session", "Generate handoff and continue in a new session", "Drop heavy content in place, recover via artifact", Snapcompact as mentioned
I dunno, I didn't read in-depth. Hopefully you don't gotta zoom in with human eyeballs.
What's helpful about that?
This is one of the most ridiculously awesome findings this year, that changed my mental model of how foundation models work.
It sounds crazy, but it's way cheaper and faster for multimodal models to read text on images than the equivalent text represented as tokens. Text tokens take a lot more bytes to represent internally than you would think. Give it images with rasterized text, and there is no OCR-like intermediate step that turns images into text tokens. It just goes straight into the multimodal context where it's all just linea algebra underneath. The cost is more lossiness than if you gave it the same tokens as text. Which is why you don't just render text prompts and files to read as images.
But compaction is inherently lossy, a summary 1-10% of the original length is going to lose things, by definition. So you keep the things you really need to get exactly right in text token summary, but put a lot more of what you would have just discarded into a PNG. Keeping the past context as text in an image is way less lossy than keeping a text summary of that context.
They peeked inside the layers and found the same neurons were usually activating in the middle layers when fed either text tokens or images of that same text. It just takes the model more time and kv cache to do it via text tokens.
Deepseek was the first to really jump on this: https://github.com/deepseek-ai/DeepSeek-OCR/blob/main/DeepSe...
[dead]