I've been diving towards this outcome for awhile now.

I have a gpd pocket 4 for my machine, but carry a chiri CE 5x3 and the MS arc mouse (and am looking for a second screen).

It's an extremely small footprint in my bag, and i'm not sure it can get much smaller. You could remove the keyboard and the mouse from the Pocket 4, but given they're on top of the hardware it wouldn't save that much space.

I can, in theory, do the same setup with my phone instead of the pocket though. I'm yet to really hook it all up and test (I expect several points of failure given past experiments), but the idea really is intriguing.

It does however require people to get more comfortable with smaller keyboards/mice (please for the love of god, if nothing else, swap left control with caps lock), or at least more portable ones.

And as for the ideal of "carry a drive, hook up to hardware as needed", that'll always run into the common issue of who is maintaining the hardware. We need cheap and easy to fix/replace hardware for that to ever really be a thing.