> ... it chains several bypass methods per request
I'm curious what are the techniques? I'm familiar with spoofing yourself as a Googlebot, or extracting the content directly when paywall enforcement is done in client-side JS only. But surely you need more tricks than that to get around WSJ or NYTimes. Also, ironically, I assume archive.is's CAPTCHA would need bypassing to fall back to that. (but volume might be low enough you can do it with a few cookies you've established trust on)
Good guesses. Crawler spoofing and pulling content that’s only hidden client-side are both in the mix, but you’re right that they aren’t enough for the bigger sites. For what it’s worth, NYT works reliably right now and WSJ is hit or miss, so WSJ is still the hard one.
I’m going to stay a little vague on the rest, because anything I describe in detail here is something a publisher can patch by Monday. The general idea is to treat each site as its own problem: several methods per request, a check on whether the article actually came back (not just a teaser), and on to the next method if it didn’t.
On archive.is: yes, the CAPTCHA is the annoying part. Volume is low enough that it’s manageable, and it’s a last resort rather than the main path.
Is it fair to say that Crawler spoofing only works on sites that don't care?
Google recommends [1] checking the reverse DNS hostname of the source IP. I don't see any room for tricks there. So if you spoof a whitelisted Crawler simply by choosing the right values for some request headers, it means the publisher's web team is either clueless or doesn't care enough, no?
[1] https://developers.google.com/crawling/docs/crawlers-fetcher...
News site probably want to be accessible to all search engine crawlers, not just Google, and I bet some of them don't have the reverse DNS verification method.
But also, tons of news sites also only implement the paywall in client-side JS! This is so obviously easy to bypass it seems intentional, and they just don't care about the 'hackers'. Or perhaps they've know that people that are going to use tricks to access an article are very unlikely to pay for a subscription anyway.
(I wish I could just pay $1 on demand and read a paywalled article. I'd do it all the time, but I'm quite reluctant to sign up for any forever recurring subscriptions)