This seems so… obvious? How can a company of this size, with its talent and expertise, not have standardized tests or specs preventing such a blatant flaw?
This seems so… obvious? How can a company of this size, with its talent and expertise, not have standardized tests or specs preventing such a blatant flaw?
First of all, Google is a shell of the company it used to be.
That said, I’d actually argue there’s an evolutionary explanation behind this where at a certain size, and more importantly complexity, an oversight like this becomes even more likely, not less.
Another takeaway: if Google can become a shell of what it once was (in terms of institutional competence, I assume you mean; Alphabet market cap seems to be doing just fine), so can your organization. As such: making something that isn't supposed to be part of your security strategy, look like it could be, is actually a long-term security risk. Sooner or later a new team will not read your own documentation, and jump to wrong conclusions. Also, it probably trains a bad security posture into your users. How many inexperienced devs saw that it was safe and expected (and apparently even required) to leave these keys out in the open, and concluded that the same logic might apply to someone else's API keys?
I think this was much less likely to happen without the needless obfuscation. If the only purpose is to identify what project the data is for, and you're trusting the client to report that value, and counseling the client to use that value in a way that trivially exposes it to everyone... what is the point of making it look like cryptic garbage? Just use the account signup name or something, and don't call it a "key" in your query parameters. Keys are supposed to unlock stuff. A name tag is not a key.
A thing I’ve learned about market cap in tech recently is that actually very little needs to get done on the core product. The momentum behind the brand is what carries the stock through time. The brand becomes its own compounding monetary instrument. Google had built a very very strong brand over the last 25 years or so. Only now is that starting to shift away from them. Because of that, I think we’ll start seeing them take more bold risks or they’ll be crushed by the weight of their own bureaucracy. This also tends to be the same reason startups can disrupt so swiftly.
An oversimplified version is this: So there are two core very critical components to the mid/late-phase tech megacorp strategy, you need to protect the core money printing product at all cost first and sustain that fiercely over a long period of time (decade+), then use any and all profits to find/fund the next cash cow, looking for optionality. While doing that, grow the market or consume a larger share of market. Google benefited from mainly the latter two and all while the internet blew up globally, funneling even more money into the machine.
It’s no secret that nearly every Google product that wasn’t search, lost them money. They were searching for the next big thing. They likely were some of the first to see AI as exactly that but moved too slowly to commercialize. Likely because of bureaucracy risk and also perhaps some sense of altruism in knowing the cataclysmic impacts AI could have. There have been plenty of former Google employees confirming this.
They also used to do things just to be cool, but those days have been long gone since Larry Page tapped out (and probably a few years before that, about a decade). Since then they’ve almost completely lost sight of what made them so successful that nobody even knows their vision or identity as a company today. These don’t correlate to market cap but they do silently lead to stagnation.
Their brand protects them from quite a lot but it’s not invincible.
YouTube and AdSense are both extraordinarily profitable. Google Play and Cloud are great revenue drivers as well. Search Ads are still the king of the hill, but it's definitely not their only profitable project, and many of the "unprofitable" projects funnel traffic to profitable ones--Chrome gives Search uplift, Google Play can't exist without Android, etc.
Definitely, they found another cash printer in Youtube but to be fair, they bought that. My comment wasn’t intended to be definitive. Yes, a lot of their other products are really just funnels to their search ads though. Strategic plays have value too of course.
They bought YouTube well over a decade ago in order to jumpstart their position in streaming, once Google Video failed to take hold, and nowadays the original YouTube would be unrecognizable. It's not like they didn't have the chops to build it, or the business acumen to make it profitable; they just acqui-hired a competitor in the space.
Any major incumbent like Google can survive for a very long time as a shadow of its former self, like Oracle/IBM/HP. Perhaps no longer in the top-20 of the S&P500, but still going as a reliable, boring, ever-stumbling blue chip.
Seems like they ought to be dedicated security teams monitoring for exactly this: does a key to X give users access to not-X. Even more bizarre is their VDP team not immediately understanding the severity of the issue.
And slow down the time to ship things? The shareholders wouldn't like that.
Those poor poor institutional shareholders…
They do have dedicated teams for exactly these sorts of concerns. They are also swamped with projects and so they can't review big new changes overnight. Google is very likely shipping first and asking questions later.
"Don't worry, we have Gemini looking at this very issue right now for all teams"
"I know, I'm reading along!"
That's how you slow down development to a crawl
Yeah, lets just start building a house and don't wait for architects to finish the blueprints :) They just slowing us down with all that thinking things through stuff.
I don't see a problem with this. The problem with "move fast and break things" isn't the moving fast part, it's the trail of broken things that no one bothers to fix. When those broken things affect people's wallets, that's when we have problems.
That's fine. Right is better than now.
I'll riff off this and say that even Google in its heyday was strangely uneven from product to product. Some products were amazing, still pretty dang good. Some products were released in a mess, abandoned nearly from the start, or driven into the ground with seemingly very little competence driving them. It always felt like Google had a bit of a darker side lurking as far as just getting basics wrong product to product / team to team.
Ah yes, I too remember Google Wave.
I don’t see it.
Imagine for a moment the there is no oversight. Every intern can ship prod code with their own homemade crypto.
How do you, in a retail business, agree to accept credentials that anyone can mint for free?
I mean obviously it happened. But… this doesn’t even seem like a compliance mistake. It’s a business-level mistake.
If you've never worked in a large corporate environment you don't know how stupid things become. In a perfect bureaucracy nobody thinks.
> In a perfect bureaucracy nobody thinks.
This resonates so well and I love it. I'm stealing this
I work at a Fortune 10.
Things get stupid for sure. But I have never once seen “hey let’s do away with access controls for high-COGS services”.
It's never that explicit, it's more the things that nobody takes care of, because it's nobody's job. The bigger the company, the more jobs fall through the cracks, that should be taken care of, but lack an explicit role in the hierarchy.
There's usually a small handful of people that care more than they should, keeping the company afloat, but it's despite the company's policies, not because of them.
I feel it in a smaller but forced growing organization as the combination of atomised responsibilities and confused/overloaded coordination. For - a certian kind of - efficiency people are isolated into their responsibility area that they are able to oversee/comprehend - with accountability - that a manegement layer is supposed to coordinate. If the mangemenet layer is now overloaded or poorly executed - confused in case of evolution and growth and any kind of restructuring - but the atomic responsibility areas are having basically no (other than anecdotic employee chatter) oversight then troubles, even obvious ones, go undetected.
> First of all, Google is a shell of the company it used to be.
Isn't that squarely at odds with Google's supposed AI prowess? Is the rot really so severe that their advances in AI (including things they've yet to make public) are insufficient to overcome it? Or are the capabilities of Gemini and AI systems in general being oversold?
> Or are the capabilities of Gemini and AI systems in general being oversold?
I pretty much sure that if anyone asked Gemini "Is it good idea to retroactively opt-in new services into for old API keys?" it would suggest it's bad idea. Problem is that no one asked.
I've learned repeatedly that LLMs are very susceptible to helpfully giving you the wrong answer when you're asking the wrong question, or asking it in the wrong way.
… Of course they are being oversold.
But also, I don’t think even Google would claim that their LLM stuff can solve problems like this.
Stuff like this was proposed to be added to standard interviews, but they were too busy reversing binary trees
Google does have a security review process on literally everything it launches.
Which is what makes this so notable. Did the security review not catch this, or did they choose to launch anyways because it was too hard to fix and speed was of the essence?
I'd expect the security team to realize what the code is treating as a secret isn't actually secret.
But there's a second insight that seems tough for a security review to catch. You have to realize that even though you can't do anything obviously malicious with the API, there is a billing problem.
Have you been on these reviews? The idea that the review will catch a misuse of the key generation infrastructure is a bit over the top.
Maybe the experienced security reviewers were laid off.
Security. The final frontier. Where no developer has ever bothered before.
To boldly allow to go where many have gone before (but shouldn't have been able to)...
in a company of this size ... left hand don't know what right hand is doing
Their “talent and expertise” is mostly in selling ads.
They probably used the in house AI tools to build this.
"This seems fine"