It seems like you mean to use your own agent (where you control the prompts), not local inference. A custom agent works just as well with cloud based models.