There's a table on the HF page that compares it against Unsloth's UD-Q4_K_XL and IQ2_XXS: https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf#fu...
There's a table on the HF page that compares it against Unsloth's UD-Q4_K_XL and IQ2_XXS: https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf#fu...
Oh, wow, they think it's just a smidge below the q4? That's crazy good if true.
The benchmarks they chose are rather cherry picked to not include long context or difficult ones that involve long horizon work or many agent turns, as I suspect this is where the model shows more differences compared to the full fat one
Yep more hops from the lower Q is likely going to skew the vectors further over time.
I wonder if there's a way to mitigate this by running it through an original Q8 draft model, attuned somehow for the PTQ1 quant, but giving it a higher threshold for the acceptance linear with the context length itself?
The longer the context, the higher the multiplier on the threshold, and more likely the draft result is used. Not ideal but it may extend the usable max context.
This model might, even without this, be amazing for short lived agents that work via generations / have changing tasks.