> I’m not really sure, I can’t say I trust any kind of synthetic AI benchmarks.

Fair enough but what exactly were you thinking when you said:

> I would like to see this (and any model router, frankly) benchmarked against two things

And thanks for sharing about failure cases! Those do sound like strange harness-level things, honestly we haven't seen failure cases like that crop up in our own usage & testing.