It's primarily relevant because it makes scraping search results much more expensive, solidifying Google's effective monopoly on Internet search.
Google has previously tried to prevent scraping of search results using legal means, but courts correctly think that scraping of Google's search results should be legal, just as Google's scraping of the whole Internet is legal. This is Google's reaction to that.
This is the best explanation. They’ve been doing the same in Google News. Each entry comes not with a URL to the source, but with a hash. To resolve it, you must send requests to Google’s servers. Anyone who wants to create a list of URLs of sources automatically can therefore be blocked by Google now on two levels rather than one - the search for a list of results, and identifying the source URL for each result.
In effect, they’re removing attribution from the content they quote from other people’s websites. It would be interesting to see if courts object to that. It is one thing to crawl other people’s websites and display snippets of their work as your search results when each result is properly and transparently attributed. But if the text is quoted and the source is not there alongside it in plaintext, replaced only by a vague promise that, if you ask, we may or may not tell you where this piece of content is from, that is a very different deal.
You can opt out from Google scraping you though? In theory you can opt out of anyone scraping you (if people were well behaved). Google should get to opt out of being scraped too.
> You can opt out from Google scraping you though?
You can't. Google will ignore robots.txt in some cases (e.g. "The REP isn't applicable to Google's crawlers that are controlled by users (for example, feed subscriptions), or crawlers that are used to increase user safety (for example, malware analysis)").
robots.txt is just a suggestion that Google loosely follows.
Well just as a regular user, I think it is pretty annoying because if I look up anything on Google while in Incognito Mode and hover over a search result, I can see that maybe the top result is maybe Wikipedia, or Instagram, or some other less-known website depending on what I'm searching for. Now, that's all very obfuscated because I don't actually know where I'm going to land for sure.
I copy a tracking link from google in my reply here. You click on the link. Google now connects me and my search session to you and knows where I gave you this link from. If I get the original url, all google knows is that I copied that link and nothing else.
They were already tracking everything you click but for example if you want to send a link to someone you can't copy the link from the Google result and send it to them, you'd either send them the Google tracking link or go to the website yourself.
It's to make it more difficult for scrapers. Google have been able to do all the tracking they've wanted for decades.
Think about it: if you want to set up something like jina.ai you need to build your own index or piggyback on Google. My guess is they use residential proxies to fire requests to Google then scrape the URLs.
Opaque URLs now give Google another gateway to detect circumvention of their anti-bot controls, helping them monetise their search index rather than allowing providers like jina.ai to succeed.
The security implications of this are very severe when you consider the amount of people who google government websites, banking, crypto and others. And google will happily serve you a phishing website either in ads or results.
it sounds like it primarily matters if you are a customer of this company, one that is building a search index off of urls scrapably hardcoded (or at least so as to be easily unencodable in non-realtime, it sounds like?) inside google search result redirects. in theory, there could be noticeable consumer user impact, but ... it would have to be a pretty large theory
> the reasons are so they can track who you are and sell your profile advertising.
What? Like they weren’t doing this before? Obviously Google’s telemetry is tracking every link you click regardless; there’s no extra tracking benefit to this.
The reason they’re doing this seems to be to stop competitors from scraping their search results.
It benefits on the other end, if I share the link and you click on it, Google knows you got it from me and from where. People 100% don't click through to share original URLs.
The fact they can track which link you clicked on is extremely valuable. For example by examining search queries and knowing which link you tried last, google can get a better idea of what is of interest and value to a user.
It's primarily relevant because it makes scraping search results much more expensive, solidifying Google's effective monopoly on Internet search.
Google has previously tried to prevent scraping of search results using legal means, but courts correctly think that scraping of Google's search results should be legal, just as Google's scraping of the whole Internet is legal. This is Google's reaction to that.
This is the best explanation. They’ve been doing the same in Google News. Each entry comes not with a URL to the source, but with a hash. To resolve it, you must send requests to Google’s servers. Anyone who wants to create a list of URLs of sources automatically can therefore be blocked by Google now on two levels rather than one - the search for a list of results, and identifying the source URL for each result.
In effect, they’re removing attribution from the content they quote from other people’s websites. It would be interesting to see if courts object to that. It is one thing to crawl other people’s websites and display snippets of their work as your search results when each result is properly and transparently attributed. But if the text is quoted and the source is not there alongside it in plaintext, replaced only by a vague promise that, if you ask, we may or may not tell you where this piece of content is from, that is a very different deal.
> It would be interesting to see if courts object to that.
It is also a measure of how enshittified and exploitative thing have become, that we look towards litigious copyright holders for assistance...
You can opt out from Google scraping you though? In theory you can opt out of anyone scraping you (if people were well behaved). Google should get to opt out of being scraped too.
> You can opt out from Google scraping you though?
You can't. Google will ignore robots.txt in some cases (e.g. "The REP isn't applicable to Google's crawlers that are controlled by users (for example, feed subscriptions), or crawlers that are used to increase user safety (for example, malware analysis)").
robots.txt is just a suggestion that Google loosely follows.
https://developers.google.com/crawling/docs/robots-txt/robot...
Well just as a regular user, I think it is pretty annoying because if I look up anything on Google while in Incognito Mode and hover over a search result, I can see that maybe the top result is maybe Wikipedia, or Instagram, or some other less-known website depending on what I'm searching for. Now, that's all very obfuscated because I don't actually know where I'm going to land for sure.
Tracking. Tied to use account, length of stay, how many reclixks, etc for marketing by/ads and surveillance.
But what’s stopping Google from doing this already?
I copy a tracking link from google in my reply here. You click on the link. Google now connects me and my search session to you and knows where I gave you this link from. If I get the original url, all google knows is that I copied that link and nothing else.
JavaScript being disabled - Google was already sending analytics pings when search result links were clicked on, using JS.
<a ping=""> works without JS in Chrome and Safari (but not Firefox) https://developer.mozilla.org/en-US/docs/Web/API/HTMLAnchorE...
Even with JS enabled, those ping requests (and other tracking) could be blocked; this new measure can not be.
https://news.ycombinator.com/item?id=49680338
JavaScript is required to use Google search at all, no?
Oh that is true, I forgot they changed that.
It doesn't work without JS though.
Pings are blockable in Firefox and so are the event analytics requests triggered by Google's on-page JavaScript.
They were already tracking everything you click but for example if you want to send a link to someone you can't copy the link from the Google result and send it to them, you'd either send them the Google tracking link or go to the website yourself.
It's to make it more difficult for scrapers. Google have been able to do all the tracking they've wanted for decades.
Think about it: if you want to set up something like jina.ai you need to build your own index or piggyback on Google. My guess is they use residential proxies to fire requests to Google then scrape the URLs.
Opaque URLs now give Google another gateway to detect circumvention of their anti-bot controls, helping them monetise their search index rather than allowing providers like jina.ai to succeed.
If you make money trying to resell Google's free services and data, this is a disaster.
As a regular user, it's a minor annoyance.
The security implications of this are very severe when you consider the amount of people who google government websites, banking, crypto and others. And google will happily serve you a phishing website either in ads or results.
Can you elaborate?
When you search Chase Bank and get a link that says Chase Bank you can't know if it's Chase Bank until you click on it
Sure. And why does that matter?
Doorway effect, once you click on it you forget you're not sure it's the real Chase Bank.
You can see the URL address bar.
it sounds like it primarily matters if you are a customer of this company, one that is building a search index off of urls scrapably hardcoded (or at least so as to be easily unencodable in non-realtime, it sounds like?) inside google search result redirects. in theory, there could be noticeable consumer user impact, but ... it would have to be a pretty large theory
[flagged]
It should be obvious, they are doing this for a reason to benefit themselves.
As others have elaborated, the reasons are so they can track who you are and sell your profile advertising.
> the reasons are so they can track who you are and sell your profile advertising.
What? Like they weren’t doing this before? Obviously Google’s telemetry is tracking every link you click regardless; there’s no extra tracking benefit to this.
The reason they’re doing this seems to be to stop competitors from scraping their search results.
It benefits on the other end, if I share the link and you click on it, Google knows you got it from me and from where. People 100% don't click through to share original URLs.
The fact they can track which link you clicked on is extremely valuable. For example by examining search queries and knowing which link you tried last, google can get a better idea of what is of interest and value to a user.