That's interesting idea. If we put into CLI with some ANSI escape code, that may become something real. Imagine a normal terminal app just render part of the UI in web and communicating in UNIX socket. While doing the fancy UI, everything is still controllable with keyboard, and optionally with mouse. The UI will fallback to text UI for older terminal
If your UI is not fully controllable with a keyboard, the same forces that made that happen will eventually make a mouse mandatory for this hypothetical tech stack too.
The terminal has no Platonic quality of being keyboard only. It is an accident of history and the limitations it has had. Remove the limitations and remove the accident of history and you will just end up drawn into the strange attractor of GUIs, warts and all.
There could be a brief honeymoon where the tech stack looks like some of you are imagining in your heads, but it would only last as long as it wasn't used by very many people. Google "gemini protocol" for a similar situation. That protocol has basically a cap on how popular it could possibly get before it just turned into HTTP B as the rest of the world forcibly upgraded it regardless of what the core project thinks. They exist in the shadow of HTTP, as the terminal exists in the shadow of GUIs. This is not a bad thing. It is what lets them be what they are. The shadows of GUIs or HTTP is large and there is plenty of space to be. Trying to give the terminal more GUI capabilities is like trying to give Gemini more web capabilities; you'll just end up in the same place, only with less refinement.
So, uh... X11? VNC? RDP?
No no not something on top of the UI stack. They also need framebuffer support so they are big headache to setup on headless server.
What I mean is that we can bring some web tech to terminal natively. We don't even need a separated shell. Security and bi-directional communication is built by default because of UNIX socket. But we still need to think how to handle stuff like cookie, local storage, external CSS / JS, ...