Thanks so much. 27b is a favourite model and last night my dev ran some benchmarks to achieve 133tks about 3.2x which changes the economics.
Thanks so much. 27b is a favourite model and last night my dev ran some benchmarks to achieve 133tks about 3.2x which changes the economics.
great to hear, which quant + KV quant + KV size are you on? is it llama.cpp?
Here's the writeup from the experiments that claude did: https://claude.ai/artifact/X6WJVtKdL1LrhYuCMqdXvG#b9a4d3e2-8...