In your blog you say:
> If every method fails, fall back to an archived copy from archive.is.
What do you reckon is the reason that archive.is is most reliable? I've seen the same in the Magnolia1234 plugin. If all fails, fall back to archive.is. What are they doing that you or the Magnolia devs can't?
I remember awhile back, people started noticing that archived pages of Reddit* had a 'logged in' status and showed the account name Archive was using to scrape the data. Not sure if this is how it works with other sources.
*I think it was Reddit.
That would be quite the oversight, considering reddit didn't require you to be logged in (awhile back). But if you happen to have a source on this, I'd be interested.
Personally, I also suspect that archive.is is simply paying for subscriptions as a last resort, but I specifically wanted to know from OP. If they do, they do remove the "logged in" status. It always appears as "logged out" in the archived version. I wonder how far they go, whether they scrub watermarks and other identifiers.
Found it! But it was GitHub, not Reddit: https://news.ycombinator.com/item?id=37528682
The content is likely filled from some other sources like Android/iPhone APIs, not from the pages themselves and then merged with page's design
e.g. https://archive.ph/8ZDI7 a page removed from original website, archive.org has paywall, but archive.is is somehow able to "archive" it paywallless
I heard someone say here that they spin up a full, non-headless browser for each site. I think they definitely pay / have accounts for some sites. That guy deserves so much respect it's insane and no what he did was not that bad and his reputation is not ruined.
They are 100% paying for a ton of different logins. How it's truly funded, who knows? It has to be using a decent ton of bandwidth too.
I don't know but suspect archive may have user login details for some.