you might enjoy learning more about this https://zed.dev/acp in case it sparks new ideas

ACP helps with agent interoperability and helps a lot of IDEs and ADEs integrate all the major harnesses seamlessly

zed https://github.com/zed-industries/zed orca https://github.com/stablyai/orca/tree/main/src/main

I haven't looked at the cli implementation but you can get very far with integrating a harness while not invoking the model directly via API