reproducible images are one of those features where the payoff is mostly emotional until the day it isn't. we had an incident where two supposedly identical images on two machines had a three byte delta in a timestamp and it cost us an afternoon to bisect from the wrong end. boring win, but a real one.

How did a differing timestamp cause an incident in the first place? Curious.

My guess is it was the only obvious evidence of an attack.

Gill probaby already knows this but for the uninitiated: something logged in, did a thing to potentially every container, and then deleted any sign of it doing the thing.

all that's left is a single timestamp of a log or something getting deleted