I've built sth similar that acts as a mobile interface to your desktop setup via the multiplexer of your choice (herdr/tmux/zellij). https://github.com/AltanS/collie

There's also a typescript SDK to connect a durable agent/harness to the app, still experimental though.