Why is the "DNS Planner" and "DNS Enactor" separate? If it was one thing, wouldn't this race condition have been much more clear to the people working on it? Is this caused by the explosion of complexity due to the over use of the microservice architecture?
> Why is the "DNS Planner" and "DNS Enactor" separate?
for a large system, it's in practice very nice to split up things like that - you have one bit of software that just reads a bunch of data and then emits a plan, and then another thing that just gets given a plan and executes it.
this is easier to test (you're just dealing with producing one data structure and consuming one data structure, the planner doesn't even try to mutate anything), it's easier to restrict permissions (one side only needs read access to the world!), it's easier to do upgrades (neither side depends on the other existing or even being in the same language), it's safer to operate (the planner is disposable, it can crash or be killed at any time with no problem except update latency), it's easier to comprehend (humans can examine the planner output which contains the entire state of the plan), it's easier to recover from weird states (you can in extremis hack the plan) etc etc. these are all things you appreciate more and more and your system gets bigger and more complicated.
> If it was one thing, wouldn't this race condition have been much more clear to the people working on it?
no
> Is this caused by the explosion of complexity due to the over use of the microservice architecture?
no
it's extremely easy to second-guess the way other people decompose their services since randoms online can't see any of the actual complexity or any of the details and so can easily suggest it would be better if it was different, without having to worry about any of the downsides of the imagined alternative solution.
Agreed, this is a common division of labor and simplifies things. It's not entirely clear in the postmortem but I speculate that the conflation of duties (i.e. the enactor also being responsible for janitor duty of stale plans) might have been a contributing factor.
The Oxide and Friends folks covered an update system they built that is similarly split and they cite a number of the same benefits as you: https://oxide-and-friends.transistor.fm/episodes/systems-sof...
I would divide these as functions inside a monolithic executable. At most, emit the plan to a file on disk as a “—whatif” optional path.
Distributed systems with files as a communication medium are much more complex than programmers think with far more failure modes than they can imagine.
Like… this one, that took out a cloud for hours!
Doing it inside a single binary gets rid of some of the nice observability features you get "for free" by breaking it up and could complicate things quite a bit (more code paths, flags for running it in "don't make a plan use the last plan mode", flags for "use this human generated plan mode"). Very few things are a free lunch but I've used this pattern numerous times and quite like it. I ran a system that used a MIP model to do capacity planning and separating planning from executing a plan was very useful for us.
I think the communications piece depends on what other systems you have around you to build on, its unlikely this planner/executor is completely freestanding. Some companies have large distributed filesystems with well known/tested semantics, schedulers that launch jobs when files appear, they might have ~free access to a database with strict serializability where they can store a serialized version of the plan, etc.
I mean any time a service goes down even 1/100 the size of AWS you have people crawling out of the woodworks giving armchair advice while having no domain relevant experience. It's barely even worth taking the time to respond. The people with opinions of value are already giving them internally.
> The people with opinions of value are already giving them internally.
interesting take, in light of all the brain drain that AWS has experienced over the last few years. some outside opinions might be useful - but perhaps the brain drain is so extreme that those remaining don't realize it's occurring?
Pick your battle I'd guess. Given how huge AWS is, if you have Desired state vs. reconciler, you probably have more resilient operations generally and a easier job of finding and isolating problems, the flip side of that is if you screw up your error handling, you get this. That aside, it seems strange to me they didn't account for the fact that a stale plan could get picked up over a new one, so maybe I misunderstand the incident/architecture.
It probably was a single-threaded python script until somebody found a way to get a Promo out of it.
This is Amazon we’re talking about, it was probably Perl.
This was my thought also. The first sentences of the RCA screamed “race condition” without even having to mention the phrase.
The two DNS components comprise a monolith: neither is useful without the other and there is one arrow on the design coupling them together.
If they were a single component then none of this would have happened.
Also, version checks? Really?
Why not compare the current state against the desired state and take the necessary actions to bring them inline?
Last but not least, deleting old config files so aggressively is a “penny wise pound foolish” design. I would keep these forever or at least a month! Certainly much, much longer than any possible time taken through the sequence of provisioning steps.
Yes it should be impossible for all DNS entries to get deleted like that.