> The switching costs of changing LLM providers is as low as it gets
Not trivial, you would need to do lots of evals and prompt tuning when you switch models.
imagine what happens when you optimize your agent skills to the current model, and new model starts breaking. you would need to have versioning for your skills, serving different skills based on the model while you do A/B testing
> Not trivial, you would need to do lots of evals and prompt tuning when you switch models.
Couldn’t we just train smaller models to “translate” what the harness user wants to what the worker model expects? I mean, if models understand caveman, it seems like just a small stretch