I mean can’t you have a grammar on both ends and just set out-of-language tokens to zero. I thought one of the APIs had a way to staple a JSON schema to the output, for ex.
We’re making pretty strong statements here. It’s not like it’s impossible to make sure DROP TABLE doesn’t get output.
You still can’t predict whether the in-language responses will be correct or not.
As an analogy: If, for a compiler, you verify that its output is valid machine code, that doesn’t tell you whether the output machine code is faithful to the input source code. For example, you might want to have the assurance that if the input specifies a terminating program, then the output machine code represents a terminating program as well. For a compiler, you can guarantee that such properties are true by construction.
More generally, you can write your programs such that you can prove from their code that they satisfy properties you are interested in for all inputs.
With LLMs, however, you have no practical way to reason about relations between the properties of inputs and outputs.
And also have a blacklist of keywords detecting program that the LLM output is run through afterwards, that's probably the easiest filter.