I kind of do this, semi-manually when using the web chat UIs (which happens less and less). I basically never let the conversations go above two messages in total (one message from me + one reply, since the quality of responses goes down so damn quick), and if anything is wrong, I restart the conversation and fix the initial prompt so it gets it right. And rather than manually writing my prompts in the web UIs, I manage prompts with http://github.com/victorb/prompta which makes it trivial to edit the prompts as I find out the best way of getting the response I want, together with some simple shell integrations to automatically include logs, source code, docs and what not.