The industry's approach to "trial and error to learn the task" is to have warehouses of robots perform various tasks until they get good at them. I imagine that you'd rely on warehouses less once you have a real fleet of robots performing real tasks in real world environments (and, at first, failing in many dumb and amusing ways).

Robots can also react much faster than 300ms. Sure, that massive transformer you put in charge of high level planning and reasoning probably isn't going to run at 200 tokens a second. But a dozen smaller control-oriented networks that are directly in charge of executing the planned motions can clock at 200 Hz or more. They can adjust fast if motor controllers, which know the position and current draw of any given motor at any given time, report data that indicates the grip is slipping.