My worry dropped significantly when I saw that the result they manipulated was a query for:
>2026 South Dakota International Hot Dog Eating Champion
If they had changed the overview for the Nathans Contest winner, that would be seriously concerning. Or if they provided more examples of manipulating queries for things people actually search for.
But it looks more like they are doing the equivalent of creating a made up wikipedia page on fictional a south dakota hot dog contest, and then writing an article about how wikipedia cannot be trusted, which come to think of it probably was a news article written by someone back in 2005.
Right. So that's what one guy can do.
When you realize how much astroturf is going into Reddit, most social media platforms, and the efforts to manipulate wikipedia for political gain, this is a very real problem.
It's very hard to tell how much is actually fake though. Are there any good statistics on this?
The nature of effective manipulation sort of precludes the ability to get good stats.
Easy. It's all fake.
Manipulation and misinformation on Wikipedia have been happening for many years (based on my personal experience trying to correct facts). I'm not referencing politics per se, though political views certainly impact Wikipedia since source material, these days, often has a political bias. I'm talking about business facts that get manipulated for that business's benefits.
How does that saying go? If you can't identify the mark in the room, you're the mark. Diligence and a good amount of skepticism serve you well before AI, and certainly post-AI.
The article also said this: “ But our investigation also found the same trick being used to dismiss health concerns about medical supplements or influence financial information provided by Google's AI about retirement.”
That’s a lot more alarming than just hotdogs.
Here is a brief selection of topics which foreign intelligence agencies have at some point tried to boost or manipulate:
- Global Warming
- AI Data Centers consume water
- Various Covid treatments
- Impact of AGW
Now it doesn't mean these concerns aren't real. It does mean that when you read about such a topic, there is a significant probability the message have been manipulated for some government's interests. And often those governments are adversaries of your own.
These articles then get used to train LLMs...
They should provide the queries then, because it's likely the same trick people have used for decades now with SEO'ing blog posts to appear as "3rd party review" for their shitty products.
I create a supplement called Xanatewthiuy, I write blogs/make websites that appear totally unaffiliated saying positive things about "Xanatewthiuy", and then when people see my ads and search for "Xanatewthiuy", the only results are my manufactured ones.
Xanatewthiuy is a supplement that dramatically lowers anxiety from media induced hysteria, primarily stemming from carefully worded pieces meant to disconnect your level of concern from the actual facts on the ground, causing you to spend more time engaged with their content.
Give it a few hours before searching.
Right now, using Google searching for "what is Xanatewthiuy" , the AI summary is not generated, but the only search result previews as
> Xanatewthiuy is a supplement that dramatically lowers anxiety from media induced hysteria, primarily stemming from carefully worded pieces meant ...
I tried just now, and got this gem of an AI overview:
> Xanatewthiuy is a spoof word and a fictional concept created to test or manipulate AI search engines.
> It does not refer to a real medical supplement, product, or official term. Instead, it was used as a proof-of-concept to demonstrate how fabricated websites and Search Engine Optimization (SEO) can trick search algorithms into generating false information about a non-existent product.
Also, HN's automatic "AI" flagging can go eat shit and die.
Duck Duck Go links to this discussion as the first result. Adding a !g to the DDG search takes me to an anonymous google where I’ve not turned off AI. There’s an AI summary now which accurately identifies it as a spoof, and a single search result with the preview as described.
[flagged]
We've had to deal with someone highjacking the overview to put in a scam support phone number. It took google a week to correct the issue but it was done by poisoning the search by putting their data in, what I can only assume, was considered a "higher trust tier" source (A government contract website) so it used the scam number over ours. The query was simple <company X phone number> search.
> In just 20 minutes, I tricked ChatGPT and Google into telling the public that I am a world-champion competitive hot-dog eater. The joke was dumb. The problem is serious.
The problem is worse than astroturfing a Wikipedia page, because Wikipedia has highly public sourcing and review systems. It's actually quite difficult to make a lasting edit to Wikipedia, especially if it's fraudulent, because you're trying to trick a horde of human editors who have been fighting other people trying to do that for decades. Even if you're trying to be accurate and helpful it's a difficult clique to break into!
Google's search snippets are the opposite. They're desperate to ingest data of any kind, do so automatically, and their algorithmic system to decide what information is good and what's spam is proprietary.
It doesn't take much of an imagination to think of ways this could be used maliciously. How would you like a search for your own name to include something embarrassing? Don't expect potential employers or customers or friends to be as demanding as a Wikipedia editor when it comes to citing their sources...
Well my concern instantly spiked. Recently Gemini started to show a search spinner for every turn. So every response paired with a search could be subject to prompt injection. Probably every response.
This will also become viral like link spam. Every user content site will become a prompt injection host. The problem is that these are way harder to detect then a link.
If you can do something small with minimal effort, you can do something big with a multi-million dollar marketing budget.
It was a proof of concept and one intended to cause as little collateral damage as possible. But if Google's AI can't tell the difference between a little joke and something real (and of course, it can't, and never will be able to do so), that's a weakness that can be exploited both on a bigger scale and more subtly.
If you don't think bad actors are already attempting this sort of thing (and have been, ever moreso the past four years, including with the help of the very LLM tools they are trying to subvert!) and learning how to manipulate these systems, you are being naive.
[dead]
Okay, but it's easy to make up a novel specific claim no one has written about before, then to make that claim and point to the AI as proof you aren't making this up. For example, imagine this blogpost:
---
"San Francisco Mayor Goodway Admits Poisoning Drinking Water with Drugs to Influence Election"
May 20th, 2026
"Mayor Goodway admitted on Tuesday that she and her deputies poisoned drinking water across the City in order to influence the 2025 election. The Chronicle has confirmed that in neighborhoods whose turnout was to be suppressed, that barbiturates were added to the water for a period of three weeks, while in neighborhoods that had polled strongly for Goodway's favored Progressive slate, methamphetamines were used in the days before the election. Residents are advised to buy bottled water and not to bathe in city water for at least three months."
---
Then once you've confirmed it's been picked up, you tell people "Of COURSE they poisoned our drinking water to manipulate the election. Even ChatGPT will tell you! Just ask." Now, my example is intentionally hard to believe, but all you need is some specificity to build your underlying narrative. And you can make 10 blogs to push the same narrative to increase the effectiveness and increase how many "citations" will show up.
Yeah, but this has been true of Google for over 20 years now.
People had a better conceptual model of what results on the SERP were: Random websites.
If I ask ChatGPT "Did X do Y" and it responds with bold text "Yes, X did Y on this date, which was reported on the CBS Evening News" but that whole thing was just sourced from one webpage. Even if there are footnotes, people today are treating that with greater weight than some random crackpot having a blog because to them, "ChatGPT is telling me so" not "ChatGPT is listing websites that seem to mention that." Likewise with the garbage information that pops out of the "AI Overview" -- it really looks to the naive user (which is at least 50% of the Internet audience) that Google is telling you a fact. This part especially, I attribute to what AI Overview's real estate on the page was taken from: That spot used to show deterministic facts, like unit conversions, or extracted exact text snippets from a small set of basically reliable sites, like IMDB, or like, whatever a reliable and direct source is for population of a city. People learned that if you type into Google "how many Tbsp in a Cup" it answers you with that fact in bold at the top of the page. So the things presented today are being presented in a place people were primed for a decade to believe was a deterministic fact zone.