I'm surprised Sol and Astra are leading "Unverified assumption" metric and Fable is better there.
I run Fable as my main model with Sol as advisor that watches every turn. Fable likes to throw around assumptions that it didn't check that are simply false, and Sol always goes to actually verify them and then alert Fable it's assuming things.
I've tried reversing this pairing with Fable as advisor. It'll just sit there going "sounds good"
I suggest you flip them. The verifier role will always verify. You will see Sol making assumptions and Fable fixing them.
But I agree - Fable makes some spectacular assumptions (which are poor assumptions).
I think what matters in this case is how proactive and greedy the model is.
GPT models are extremely proactive and gredy. So when Fable mentions something that may affect some obscure component of the system, GPT will start digging the codebase, execute web searches, re-read AGENTS.md and hit fable on the head.
Fable never does that, it just reads the turns and acknowledges it read them. This also explains why GPT models tend to overengineer things and why they're amazing reviewers if you triage their findings.
GPT doesn’t do all of that all that much when it itself is the implementer. RL has made implementation and reviewing two different behavior sets.
In open weights models on a similar metric I find this to be highly specific to role and model. Some models are just diggers by nature, but all will verify more if you tell them to. Have you tried running a Claude watcher with the same role as Sol to see if it verifies as much?
What do you use as the harness?
oh-my-pi.
I was using my own homegrown (mega slop) harness for a while, but it distracted me from working on my actual projects, and I realized oh-my-pi was doing the same things I've been doing, including advisor, native server-side compaction, etc. It's a really good harness.
would love to chat and learn more about your set up! here's my email - janak@withspecific.com