> <200ms for 45 questions at once
Considering your own question length: ~120 characters x 45 divided by 4.1 ~= 1317 tokens.
So question processing at 5.5k PP(around the actual PP speed of GPT5.6 Sol) it would take around ~0.24 seconds + the context processing.
Computing the output should be around ~20ms (at 50 tok/s), computing 45 tokens in parallel.
> have 0% malformed output
Pretty trivial; only the allowed output is selectable :)
So, I keep repeating myself: Jev was a low-hanging fruit all along; no one cared, and probably no one will in a few weeks?
Yeah but a lot of developers who didn't even know that this was a possibility now do, and will probably find use cases for it.
You can probably even share context between questions by cleverly manipulating the attention mask.
Nice idea! Didn't think about that; a single linear memory allocation could do the trick