Great point and realisation on many realities of robotics. Most successful and scaling projects have been traditionally in controlled environments. This always reminds me of William Shatner in "The Future is Now" ad in 1984: https://www.youtube.com/watch?v=pb_-BWHMQlc

When we can control the environment precisely, we should as it simplifies so much---we can clip the world to platonic geometries and apply common tricks and trigonometry.

But when we cannot control the environment, really really much harder. Self-driving cars are a good example how hard it is, and still mainly focusing on relatively structured environments like cities.

Arbitrary environment is really where many expect AI/ML/probabilistic robotics to bring some necessary flexibility. And still out of reach today, except some astonishing use cases like Roombas (well, more like a best effort approach, still).