We have systems for highly interoperable and compostable GUI applications - think NextSTEP or, modern day, dbus, to a lesser extent.

Really, GUIs can be formed of a public API with graphics slapped on top. They usually aren't, but they can be.

I weep for what happened to AppKit/Cocoa

Just because it says compostable on the container doesn't mean it will actually break down in a reasonable amount of time on your home compost heap, or that they don't leach some environmentally harmful chemicals in the process.

Many modern web apps are just APIs with a browser GUI.

I'd say ROS (Robot Operating System) is the closest to this ideal.