We should expect pullbacks, fuckups, plans failing, and rollouts getting canned. It's part of how humans do things. Its actually a pretty effective optimization algorithm.

I'd bet that some sort of exponentiate the learning rate until shit goes haywire then rollback the weights is actually probably a fairly decent algorithm (something like backtracking line search).