Is it fair to say that Crawler spoofing only works on sites that don't care?
Google recommends [1] checking the reverse DNS hostname of the source IP. I don't see any room for tricks there. So if you spoof a whitelisted Crawler simply by choosing the right values for some request headers, it means the publisher's web team is either clueless or doesn't care enough, no?
[1] https://developers.google.com/crawling/docs/crawlers-fetcher...
News site probably want to be accessible to all search engine crawlers, not just Google, and I bet some of them don't have the reverse DNS verification method.
But also, tons of news sites also only implement the paywall in client-side JS! This is so obviously easy to bypass it seems intentional, and they just don't care about the 'hackers'. Or perhaps they've know that people that are going to use tricks to access an article are very unlikely to pay for a subscription anyway.
(I wish I could just pay $1 on demand and read a paywalled article. I'd do it all the time, but I'm quite reluctant to sign up for any forever recurring subscriptions)