Why client side? Doesn't that mean re-uploading the whole context whenever you change something? Seems like storing it server-side and providing an API to swap pieces in and out of context would be a lot more efficient. Or if the LLM is the one deciding what to swap, that could be done without client involvement at all. Even better would be if this API was standardized across providers.

Outside of context manipulation, it'd also be nice to standardize a format to label sections of the context and later append keywords to the context to ignore / unignore those sections (or refer back to them, emphasize them, whatever). With that, I imagine we'd be able to have a pretty good set of standard LoRA adapters that enable all LLMs to be controlled in the same fashion. That way agents will be able to manipulate LLMs in a standard way, without having to rewrite context itself.