I have often thought layouts should be done by a constraint solver. Then there could be libraries that help simplify specifying a layout, which feed constraints to the solver.

I've done that for desktop apps before. You have to be careful with the effects of sub-pixel rendering and whatnot if your math is continuous, but it's a viable path that I quite like.

Don't use continuous math in either a design system nor a constraint solver that you expect random developers to use. Either case will only lead to problems.

I largely agree, but there's a little nuance insofar as "interior-point" methods are very powerful. You can go a long way by encoding your goals as error functions and letting a gradient-based optimizer do the rest.

iOS used to do this using the Cassowary constraint solver pre-SwiftUI. It’s the worst thing to work with. So much code turning on and off constraints, dynamically adding constraints when you have new views. And that’s before you get into conflicts