Interesting. Could you not tell it to not second-guess itself (on minor issues)?

Fighting against the weights is often a lost cause. For any model.