From Google's status page https://status.cloud.google.com/incidents/Xs2XTHzAVC9YnafnPy...
We are experiencing an issue with multiple services in
europe-west2 beginning on Monday, 2026-10-19 23:08 PDT.
Due to a power event we had suffered an outage within
europe-west2-b zone. The power event is mitigated. Partial
mitigation to infrastructure services will be observed within the
next 30 minutes and the remaining services will be mitigated there
after.
(Also reporting the times in PDT is quite unhelpful for users of this zone who are likely based in the UK or EU)
I'm not saying it's the case here, but sometimes the cloud providers themselves have inter-zone dependencies, or even inter-region dependencies that can cause outages in other zones and regions.
For example, AWS us-east-1 outages have caused global outages on multiple occasions, such as https://www.networkworld.com/article/4168878/aws-hit-by-us-e...
In Google Cloud, that's a huge no no and you'll be put on a naughty list facing serious scrutiny during the postmortem review.
Disclaimer: I work for Google Cloud.
Is it possible someone just tried to serve five terabytes?
No. That is definitely impossible. Nobody gets to serve five terabytes.
Ok, but why were all these higher level services that are not zonal experiencing issues then?
Yes that's a good strategy if someone wants to get some face time with TK.
They don’t do periodic BCDR testing?!
Which 'they?'
If you mean Google: Yes, extensive. One of the techniques they use is DiRT (Disaster Recovery Testing) which is a scheduled time-frame in which Google purposefully breaks systems that are exceeding their SLAs to verify that automated recovery executes as intended. In general, a DiRT test would catch something like "This service has a key dependency on us-east-1."
Now, in the past (also a Xoogler)... Some pieces of Cloud were moving at a cadence that they got away with a bit of a relaxed DiRT awhile back, so if someone tells me "I remember when a us-east-1 outage took down a key Cloud service" I would believe it. DiRT is also planned testing, it's not slapping a chaos monkey on the cluster fabric and crossing one's fingers, so it's entirely possible for key outage modalities to get missed (engineers are human and even though Google aims for a "no haunted closets" engineering practice, people develop a sixth sense of where the weak parts are of your infra and it takes courage to commit to pushing on those until they break).
Google SRE's official (?) motto is "hope is not a strategy".
Could you explain how other regions were impacted today ?
Maybe OP is on naughty list facing serious scrutiny during the postmortem review.
And yet I had stuff in other zones that was affected.
People who don't work on cloud services are usually very sloppy with their wording which I think is leading to some disconnect here. A "zone" is not really the unit of availability for most things people consider the "cloud" for hyperscalers. A zone is like 1-3 DCs and you are very vulnerable to stuff like localized power events. For that reason most services are not offered at the zonal level and if they are you get big disclaimers saying so and lower 9s of SLA.
Most things are at a regional level which is multiple zones. Your AWS us-east-1 is one such region. A lot of datacenters spread across a decently large physical area have to have an issue for a whole region to start having failures.
For GCP it seems like one power event in a DC led to a thundering herd on the rest of the region which was the actual issue here. (which is a pretty significant problem and isn't supposed to happen)
You are generally right, except that you don't seem to know the history of major AWS outages. Failures in AWS in us-east-1, which is the first AWS cloud region, often cause cascading worldwide failures in the whole of AWS - especially failures related to DNS and to DynamoDB, if I remember correctly. That is, AWS workloada running in, say, eu-west-1 (and in fact in the whole of AWS) often start failing or at least having greatly degraded performance/availability because of a failure in AWS infra in their core region.
If AWS themselves can't handle cross region isolation and availability well, my expectations for most companies running on top of AWS are quite modest.
It's because AWS runs their control plane out of us-east-1, particularly lambda. There are also some very popular services that can only be run out of us-east-1. It's not really a technological constraint, it's a design decision.
Wait so then how are they signing big customers still who would definitely lose a lot if us-east-1 went down?
(who would be motivated to check if anything is restricted to only us-east-1)
They had a comparable problem in europe-west9 in 2023. Apparently they actually run into trouble if they lose a whole zone through a physical event. It's either thundering herd or simply insufficient capacity in the other two zones to take over all the load from the third zone.
This is why Microsoft calls it an Availability Zone, each zone is made up of one or more data centers equipped with entirely independent power, cooling, and networking infrastructure. One zone should not knock out other zones. Zones are also often nearby each other just enough that they have very efficient fiber optic connections between them.
That said, that's how Microsoft handles it, I'm not sure how GCP compares.
It is an interesting conundrum though, what if all your nearby availability zones come down at once due to power loss? I wonder which major cloud provider has the best setup for these sort of events.
>It is an interesting conundrum though, what if all your nearby availability zones come down at once due to power loss? I wonder which major cloud provider has the best setup for these sort of events
im only familiar with microsoft in this sense, but they have "geo-zone-redundant storage" which copies data across 3+ availability zones in the primary region, and also a secondary geographic region (i think at least 100 miles away if i remember, but maybe further). any incident would have to be massive to affect multiple geo-zones
Thundering heards during zonal outage is fairly common across the providers. You are substantially better off being multi-regional than multi-zonal.
Worked at a major ISP in the 90s with many critical infra customers. True diversity is a lot harder than many imagine.
> I'm amazed the big companies reported as down weren't using multiple zones
Much as with backups - if you don't test them, they don't exist.
Everywhere I've worked has had multi-AZ infrastructure. I have never worked anywhere where the failure of an AZ has been simulated regularly, with second-order effects mapped out.
I'd imagine (maybe optimistically) most companies do think about using multiple zones. And even test for DR. Regularly. And successfully. Problem is, all that works just fine on a sunny Monday morning. Then shit hits the fan and all of a sudden everyone is forced to, at the same time. It's a different kettle of fish
We set up our applications and services to be multi-AZ natively. It's not particularly difficult with how many AWS services are regional or global (Admittedly, I still expect a potential DNS failure to cause issues if and when that occurs). However we are just part of a workflow, we integrate with other applications that are not multi-AZ and would be impacted. That's just out of our hands.
It's like our on-prem systems were ordered to plan out and exercise DR plans to account for a data center going offline, but we can't exercise a DR plan fully if part of the plan is 'Adapt to how application X, which is hosted in the same datacenter, also chose to fail over.' We do not own application X. There is no appetite in the business to co-ordinate the DR exercise across the entire company at the same time.
We can't even do it all in a lower environment because there are dependencies on systems that are shared across environments. Some massive, redundant filesystem that is sync'd across our data centers, but the sync is a boolean. One datacenter instance of it is the primary and the others are read-only until they get promoted. We have one of these and lower environments are just a different root folder on the system with different access patterns, but they can't make DataCenter A the primary in prod and DataCenter B the primary in UAT.
Question who's never relied on this: how do you test whether your multi-AZ setup can withstand an AZ dropping? It sounds like your third party apps make it kind of pointless to test anyway, but I'm curious!
Ops type here, generally you have to fail an AZ and see what happens. If you have Dev/Staging environment that is exactly like production with plenty of tests, you can use that. I've seen only one environment actually like that though.
Since most companies would have to do it in production and they don't want to have customer impacting outage, they tend to do it during an actual outage.
Thans! So do you have a staging environment for everything, then cut off the network or kill the instances, Aurora DB, etc.? Simulate some load while its going on, and then check for data loss? Then if that works try it in prod?
It seems like a lot of work, and it seems that as architectures evolve it would be hard to ensure that the multi-AZ failover works well.
Yes, which means it's expensive which means companies don't usually pay for it so we should name and shame companies who aren't willing to pay that expense just to pad their bottom line.
I believe the article in question partially provides such a list.
GCP is kind of odd in that some hardware is only available in certain zones, and some hardware is technically available in multiple zones but the capacity is not evenly spread. I've had to implement infrastructure code for GCP that interrogated hardware availability by region and zone to place VMs, trading off multi-AZ availability where it made sense.
Other clouds are similarly odd (or even worse). I have experienced in Azure that if you provision a resource group with some basic SKU for a given service, the only way to later upgrade to the latest and greatest SKU is to tear down the whole resource group and re-provision everything from scratch. All attempts at scale-up in place will fail with cryptic errors. Seems like you are assigned in an old corner of the DC and can't get out of it.
Extra fun if there is a key vault involved, then you have to manually delete every key in it, otherwise you can't create new keys with the same names in the new vault.
I can't even get a managed redis instance in Azure EUS2 right now...
Yes. The area on the Venn diagram where services, hardware and regulatory compliance intersect is surprisingly small.
I had a client that was contracting with a very large bank and there were only something like four zones globally that we could possibly deploy in to.
Google also is unlike AWS in that their zone abstraction is very virtual, and two zones might be in the same physical building. If you have requirements that your deployment zones be physically separated by some minimal distance then it can make things pretty challenging.
And afaik there is no way to get an exact physical location (DC, Hall, rack) for the hardware you are actually using. If there was you could at least verify if there wasn't some kind of hiddent dependency.
> infrastructure code for GCP that interrogated hardware availability
How do you to do that ?
There's a GCP API that lists the hardware advertised in each zone as a starting point. Then you have to actually attempt to provision hardware in each zone to try to get a sense of what capacity is available there. Combine the two and you can kind of map out a topology.
From the documentation of the Google Cloud Platform.
> To protect against the loss of an entire region due to natural disaster, have a disaster recovery plan and know how to bring up your application in the unlikely event that your primary region is lost.
> The regions Stockholm, Osaka, and Montreal have three zones housed in one or two physical data centers.
See here: https://docs.cloud.google.com/docs/geography-and-regions
2023 europe-west9 incident: https://www.datacenterdynamics.com/en/news/water-leak-at-par...
>> That is why all the cloud providers have multiple zones....and I'm amazed the big companies reported as down weren't using multiple zones.
Well then you should know that a Google Zone is not the same as an AWS AZ and many of Google regions are just firewalls across the same data center.... Despite all their SREs and Books and Phds...they dont know how to do infrastructure.
For sure Google zones aren't the same as AWS AZs, and I remember complaining about this years ago during that fire in a France DC. But you also said their regions are firewalls across a datacenter, suggesting a multi-region DC, which I'm pretty sure they don't have.
The overall point is accurate. Google expects you to spread things more across zones. Most users would rather zones not share buildings.
Dont downvote...read their docs....and this is just the start:
"The regions Stockholm, Osaka, and Montreal have three zones housed in one or two physical data centers." - https://docs.cloud.google.com/docs/geography-and-regions
"This means the multiple zones within each of these regions may not be located in physically separate data center campuses." - https://docs.cloud.google.com/backup-disaster-recovery/docs/...
"A region is a set of buildings operated by Google in a geographic location, such as a data center campus." - https://cloud.google.com/vpc/network-pricing
Google is PDT-centric, and never grew out of it. Everything is Mountain View time internally and it's so jarring.
It comes across as really uncaring to the probably 99.9999% of people outside of the US who have no idea offhand what UTC offset it is, it almost feels hostile.
If they at least included “(UTC -7)” after PDT, while still a bit annoying, at least people could work out what time that is without having to lookup PDT from somewhere first (as I just had to).
Shrug. They're big enough to set a standard (much as we have GMT because France and the UK had enough political / economic / cultural gravity in the era of the steamboat to make the world care what a "Greenwich" was).
We just keep a clock on the wall set to Google time, like they do in newsrooms and NASA mission control. It's convenient enough.
MTV time is fine if it's consistent and says that it's MTV time. From what I remember, a lot of stuff at Google doesn't clarify the time zone, and there's some UTC stuff mixed in.
> It's convenient enough.
It doesn't sound like it.
You get used to it. It's exactly as convenient as having all my logs in GMT and then having to do the math to convert from those numbers to when people on the team were actually awake / asleep / at lunch, especially since my team is smeared across a half-dozen timezones too.
The annoying thing is that everyone thinks they're big enough to force their own TZ.
Southwest airlines does it.
Rackspace used to do it, to the point of needing to ask them pretty please stop provisioning hardware on central time. Obnoxious.
I'm impressed that Stack Overflow / Stack Exchange (presumably US-based) uses UTC for things like resetting daily vote limits.
And tbh, Florence Mean Time would've made more sense in lining up the world on the map.
On Mercator map projection, sure, but then we'd have to address that tyranny as well.
The fact they didn't immediately fail over doesn't mean they didn't had a plan.
The plan might've just been "if it gets bombed, we restore backups into other zone". Just waiting for it to fix itself for few hours mightv'e been acceptable risk.
And what about the fact this give the US government?
1) the ability to view everyone's bank data, in secret
2) the ability to change EU bank data at will, in secret
(see US Cloud act)
There are much worse things you can do to data than bomb it.
> https://status.cloud.google.com/incidents/Xs2XTHzAVC9YnafnPy...
https://news.ycombinator.com/item?id=50163858
Often there are irritating legal geographic limitations on where cloud hardware can exist for a given country. I can understand why, but there is this particular drawback of having all eggs baskets close together.
The drawback being temptation not to power them indepedently, it seems.
I know some banks out-sourced most of its infrastructure to external operators, and some operator I worked with didn't seem competent.
> ... I'm amazed the big companies reported as down weren't using multiple zones.
Yeah well, they drank the cloud kool-aid. The promise was that they'd go "cloud", and that everything would be taken care of.
Turns out: availability is worse than when those companies used to self-host and major cloud outages are happening all the time (at this point it's basically a meme).
And now the promise that "the cloud is simpler than doing it yourself" turns out to be not true?
I'm not surprised at all: these companies have been sold a lie.