That is pretty much what I do, although my "thin generic program" is still very beta.

I follow the narrow waist principle, and basically have all communication in the form of setting and subscribing to "tag points", plus getting their metadata.

The controllers have some ability to run logic, in the form of web editable "If you are in this state, every frame, set this variable to this expression", and they have a generic UI to set and view all the variables.

But I don't have any legacy controllers to deal with, just ESP32s.

If I do run into legacy stuff, I just leave it alone, my main controller lets you write plugins that make whatever device you want appear as one of these tag point based devices, so I keep as much of the flexibility as possible in nice version controllable python.