Going back between two different company's AI tools when facing a tricky architecture question often surfaces holes in an approach I'd been building.
Similarly, if I ever get a bit too vibey and don't carefully review code changes myself, the blast radius is generally significantly resolved by a carefully tuned "did you consider x, y, and z" skill after a first draft partnered with a "deploy an adversarial review agent for the worktree".
When I was doing some important work, I had 2 worktrees, each running a frontier model from a different company, and I passed their responses between them in markdown files and had them critique them. I thought they'd prefer their own solutions, but they frequently preferred the other model's solutions.
I went back and forth like this until we converged on a solution "everyone" was satisfied with.
---
Context: I'm a solo dev and working outside my area of expertise, so I'm leaning heavily on LLMs. It's not great (definitely too vibey for my taste, and I keep running into issues) but the alternative is spending the next few years studying several specializations instead of shipping. So this is the "least bad" thing I could come up with.
I'm definitely increasingly making time for "learning sprints" to catch up on specific knowledge gaps. For example today I spent 2 hours debugging something because I was missing a fact that would have taken me 1 minute to learn...
I went from "fully hand crafted" to "fully vibed" (when Fable came out), back to "fully hand crafted" (once I realized I no longer understand the code!), and now I'm at "making very careful use of AI, asking for the smallest possible changes, and double checking everything"...
> I thought they'd prefer their own solutions, but they frequently preferred the other model's solutions.
I suspect that's downstream from their sycophancy.
Ha, maybe. Sample size was low but usually both models ended up preferring the same solution.
I had a similar situation set up where I would have two models come up with their own independent diagnoses and plans to fix problems.
Then I would have them each read the other's plan. But I would tell them each something like this, "I had a friend look at this too. He's smart, but in general you're smarter and more knowledgeable than him, so don't be afraid to say where he's wrong."
I even tell them to give me an adversarial review and tell me what's wrong with the approaches.