Not every vector field has a potential. So not every weight update can be written as a gradient.