So, effectively if A → B → C → D and D is failing, C may retry D, but B and A are discouraged from retrying the whole chain.
This is quite slever. I also really like the concept of an "Error Budget", inspired by SRE and SLO(s) no doubt :)
So, effectively if A → B → C → D and D is failing, C may retry D, but B and A are discouraged from retrying the whole chain.
This is quite slever. I also really like the concept of an "Error Budget", inspired by SRE and SLO(s) no doubt :)
I had a former colleague who would put a “retry 10x with sleeps” in each of A, B, C, D.
None of these were even expected to fail. But the code was buggy as well, so after D being retried 10000 times, it’d eventually give up. Managed to convert a sub-second operation into a half hour affair.
LOL