This looks super clean. I'm curious about the --judge command. How does it evaluate if the cheaper model's response is a "tie" or acceptable? Is it using a specific LLM-as-a-judge prompt template?
This looks super clean. I'm curious about the --judge command. How does it evaluate if the cheaper model's response is a "tie" or acceptable? Is it using a specific LLM-as-a-judge prompt template?
Thanks. Yes, that's exactly right, and aptly named so in the code.
A few things to note: the prompt is actually deliberately tie-biased; the tie only then breaks on clear material differences: factual error, missing information the prompt requested, or just a complete failure to follow the instruction. The judge is explicitly told that response/output length, wording, style and formatting are not quality differences.
To ensure there's no favouritism, the judge only sees the outputs (A and B), it never sees whether the output is from the current model or from the candidate; and to ensure no funny business occurs, the outputs for A and B are randomized.
The default model is chosen automatically from the highest-tier model in the log, but this could invite self-bias (this is cautioned in the report output), so you can override this with --judge-model
You can check out the prompt for yourself in src/frugon/measure.py - search: JUDGE_PROMPT_TEMPLATE