You got cut off at the end, but I hear your point.

I completely agree that cost-per-successful-outcome is the goal, and "cheapest" is never the entire solution, but only routing to the cheapest isn't what frugon does. If it did that, then it would just pick something like GPT-4.1-Nano (one of the current cheapest models regardless of quality) for everything, and it would be useless.

The whole pipeline in frugon is predicated on quality tier gating. Frugon doesn't strip out quality. This is based on quality tiers from LMArena leaderboard data (CC-BY licensed). The easy/hard split only recommends routing calls predicted to survive the downgrade.

> $ is operationally useless since $ is model depenedent

> model+harness are symbiotic or antagonistic

To these points, frugon gives you --measure (you can compare the outcomes yourself via prompt sampling from both models - current and candidate/recommended) and --judge (championing a model to determine whether the recommended/candidate model actually successfully holds the quality of the current model from the prompts sampled). The win/tie rate summary outputted by the judge tells you whether the candidates you're comparing preserve the outcome you're currently getting from the current model measured on your own prompts. So, if your model succeeds at the prompt and the candidate ties this (judge's verdict), then you've successfully evaluated that you can swap to a different model at a lower cost. It's measured on your task distribution as a quality confirmation of frugon's initial offline quality tier gating routing recommendation.

Success rates are task-dependent, as every user prompts a model differently which could either get a successful outcome or a failure-doomed-to-retry outcome. No benchmark/table can tell you what your agent+subagent's odds of success are, unless all you do is pass an unaltered prompt that has been externally benchmarked. Hence why --measure and --judge exist. You must confirm the initial routing recommendation with your own evaluation (human/ai/evals)

However, you have made me think of combining the initial dollar cost analysis with judge outcome as an "effective cost per judged success" metric...which is a great idea, as they are currently separated.