"Finally, we are in planning to remove Google Cloud services from our data plane’s hot path, and keeping them only for secondary/failover."
That's pretty clear. Google can no longer be trusted as a B2B service provider.
"Finally, we are in planning to remove Google Cloud services from our data plane’s hot path, and keeping them only for secondary/failover."
That's pretty clear. Google can no longer be trusted as a B2B service provider.
Meta is no different. I know a company that had their OAuth app on Meta rendered completely unusable just because one of their employees (a dev) had their personal Facebook account banned by Meta for no reason. They tried to escalate it multiple times but got nowhere, lol. Meta is even worse because accounts need to be 'personal'; if you have a Business Manager, the users added to it are all tied to their personal Meta/Facebook accounts. This is ludicrous.
To me, building any business with dependencies on Meta is just a bad business plan.
Yeah, people loose their business because a kid is logged in on their iPad, gets their google account suspended, and google knows it's the same household as the parent, and everything gets shut down
Can't find this now but google did at least once disable company's accounts after dev got their account suspended.
And as we know from the recent Gemini ban wave, you can get suspended just because.
[dead]
Everyone needs a defensible root of trust, this goes all the way down to the registrar you use for your domain.
> google knows it's the same household as the parent,
Nearly all these linkages are due to people sharing recovery email addresses and phone numbers. Don't do that.
Are you honestly saying that a kid should not use their parent's email address as a recovery option? Seems like that would be the natural way to do it.
I don’t know about you, but I have a family account that we use as an email recovery for kids.
Adults have multiple emails so they won’t have to share it.
If something takes out the family email account, that’s fine. The only thing going there regularly are school notices, contractor receipts and recovery emails.
It’s almost impossible not to any more. This is victim blaming at this point.
Meta and Google B2B are both horrible. Their ad account bans are constant, and they have no real escalation process to get help. These companies are monopolies that should treat businesses more seriously, especially in these situations.
[flagged]
The is in context of B2B, which meta has a huge ecosystem and often rips away a companies revenue for hidden reasons
Crazy considering this was their primary argument against the App Store's revenue share model. Not that they're wrong, but you'd think they would at least be consistent.
Seems relevant to me as it is still a service that their company relied on.
Sure you're not misreading Metal?
They're a popular SSO provider.
A huge number of small businesses have no Internet presence beyond their Facebook and Insta pages, so … yes they are extremely relevant to a discussion about the risk to small business of flaky hyperscalers.
More businesses need to hear this message. Google has proven time and time again they cannot be trusted as a service provider, exactly because of this problem.
They trust them enough to still give them money, just goes to show how entrenched big tech is and why they need to be broken up into dozens of pieces.
In the meantime start by migrating away from them for anything serious.
There is a history going back many years of Google suspending or terminating accounts with no explanation, often having to backtrack after users published their frustration and the incident went viral.
Google has always acted as if they have no obligations whatsoever to their paying customers.
They have not explained WHY their account was suspended. That's the most important part, imo. Cloud Providers don't suspend entire accounts for no reason.
> "Cloud Providers don't suspend entire accounts for no reason."
Maybe I'm getting old but here[1] is a HN comment from 17 years ago complaining about Google banning accounts "by mistake" and having no recourse but to post on HN and hope Matt Cutts sees it and helps, and saying "there are literally 1000s of such stories for many years all over the blogoshphere and forums" which is something I remember from HN of years ago.
[1] https://news.ycombinator.com/item?id=791004
And unfortunately nothing has changed since then regarding this.
The cloud provider in question - GCP - who also deleted a 125 billion dollar company's entire account on accident?
What company?
In May 2024, Google Cloud Platform (GCP) accidentally deleted the private cloud account and all backups belonging to UniSuper, an Australian pension fund managing over $125 billion.
I think that stretches what it means for a company to "be" a 125b company, but that is still awful.
They are a pension fund; they literally had/have US$125 billion dollars under management. What exactly is being stretched here? I can't for the life of me think of something that qualifies more for being a 125b company than actually having 125b in assets.
Having assets under management doesn't mean you have that money. You don't own it, you are just taking care of it for somebody. When describing a company as an $X billion company, conventionally this is referring to the market cap. You could use it to describe other things they possess if you wanted to, but assets they manage will never be something they possess.
Companies are described by revenue. UniSuper made $110 million recently. It deceptive to use the assets managed as the size since it makes it look like a much larger company. NVIDIA has revenue of $130 billion. $125 billion revenue would make it the largest company in Australia by a good amount.
> I can't for the life of me think of something that qualifies more for being a 125b company than actually having 125b in assets.
Which this company didn't. They managed 125b of assets belonging to other people, they didn't have 125b of their own.
> Cloud Providers don't suspend entire accounts for no reason.
You're joking, right?
LOL, did you woke up from the hibernation?
This is Google we're talking about. This absolutely happened many times in the past and will happen again.
Cloud Providers don't suspend entire accounts for no reason.
Oh...my. Just starting out in the industry, are we? Those of us who have been here for a while know reality is very different than newbie hopes and dreams. Once you've been burned for the n+1 time, that optimism will fade.
Google has suspended entire accounts countless times for absolutely no reason.
Unfortunately the cloud providers also rarely if ever tell you the reason.
Not defending them. but wouldn't it be a legal nightmare if they did?
My guess would be the credit card expired....
If it were something out of Railways hands, I think they would say something like "We have not yet identified the reason for the suspension, and are awaiting a response from Google".
At any company doing Enterprise work, you don't cut off someone for non payment without Account Manager doing multiple phone calls to whoever you have contact information for, emailing everyone listed on the account and whoever opened a support ticket and maybe even putting a banner in the panel with "ACCOUNT OVERDUE, CALL US TO SORT IT OUT!"
Generally it takes 30 days past due and complete no contact for anyone before suspension.
No one pays $2m invoices with credit cards.
it surprised me a lot when I first encountered it, but some organisations do :)
FTA:
> Google Cloud placed Railway’s production account into a suspended status incorrectly, as part of an automated action. This action extended to many accounts within Google Cloud. As this was a platform-wide action, there was no proactive outreach to individual customers prior to the restriction.
This might be 100% of what google told them.
Never could. Google might block your entire company because one of your workers did something nasty on their personal account, and their ban hammer is mighty and blocks all related accounts to the Nth degree
I've been wondering if I should be interrogating my friends before allowing them access to my wifi. "Have you or any of your family members ever been banned by Google?"
Somebody else's computer
Hasn't every cloud provider had issues? Is the enshitification of servces really isolated to Google, or are we all doomed.
Banning accounts for no stated reason is kind of a Google speciality. They have a long well documented history of this sort of thing.
Railway has an overwhelming incentive to pin the blame on Google. This report doesn't answer why Google suspended Railway's account.
I'd wait for more details before adjudicating.
In principle, I agree with you.
In practice, Google has earned the way my priors are ready to believe it's 100% their fault with mighty and sustained effort. Or lack thereof, depending on your point of view.
That would be approximately 6365262822 time Google suspended someone for no good reason.
So no, Google doesn't get the benefit of the doubt.
They said it was automated and affected a bunch of other customers, which gives at least some hint.
And in general Google lost any immediate benefit of the doubt status many years ago. Many such stories.
To quote the article:
> Google Cloud placed Railway’s production account into a suspended status incorrectly, as part of an automated action. This action extended to many accounts within Google Cloud. As this was a platform-wide action, there was no proactive outreach to individual customers prior to the restriction.
Railway don't have a great reputation for building scalable systems (effects of vibe coding?). It's worth waiting for Google's response before jumping to conclusions. They can move to Azure/AWS/own datacenter, but there's a good chance this will repeat in a few months.
I understand this opinion, because their API keys / OAuth tokens had no permissions structure, so a user of the Railway MCP had their infrastructure destroyed by an overzealous LLM agent. However, this is orthogonal to their infrastructure capabilities.
Sure, if this was one off isolated incident people would have agreed with you. But it's not. Even Google personal accounts have been used to ban their other ones including ones spending thousands of dollars on ads or GCP or any other paid google service, which is ridiculous.
Their reputation is fine, and their uptake is due in part to their handling of scaling.
If you're picking them instead of the underlying cloud provider, but you want all the knows and dials the underlying provider has, you've made the wrong choice.
There is always one bootlicker, fresh 1 day account no less.
Been a passive reader here at HN for too long, finally registered today. Instead of viewing this incident objectively, you choose to insult me (?).
I know multiple startup founders personally (2 of them are in the current YC batch), and the sheer callousness with which they look at infra, especially from security/scalability/reliability angle is shocking.
I'll personally reserve judgement against GCP (replace with AWS/Azure/OCI/whatever) until we know more.
Then let me be the not day-one account to say Railway is utterly bearing some responsibility here.
"However, in this ring, there was still a hard dependency on workload discoverability being tied to the network control plane API that was hosted on the machines running in Google Cloud."
They've gotta be joking me that they deliberately left something so critical under the control of any other entity than themselves. That demonstrates a lack of critical planning and a lack looking at their configuration from a first-principles approach.
There is always responsibility with Railway, that's given. But also taking into account how many big websites went down when AWS was down, building critical redundancy at such large scale is not cheap, and not many companies do it. Same as security theatre, we have redundancy theatre because they needed to sell the CLOUD.
I'm not sure that's the lesson to learn from this outage. Hell Google resolved the problem in 7 minutes which is as good as you could hope for.
The resulting action should be you have proper disaster recovery, failover, etc.
Not sure I would trust these folks if this is the conclusion they are coming to from this experience. Any cloud provider can/will do this to you.
Google restored access but did not resolve the problem. VM’s were still shut down.
Google resolved the problem in seven minutes for a billion dollar company. Good luck if you are a nobody.
The best you could hope for is that if there is something fishy with your account, you are contacted by Google to address it.