So they made a prior tweak to reduce fetches of favicon.ico ( by link rel="icon" href="data:," ) but why do they still serve the overlarge meaningless html page to requests for robots.txt ?!
Given that they have gotten spiked by automated requests lately, aka agents, wouldn't this at least be respected by the big players?
Also, as they are using Cloudflare, wouldn't a discussion (or a note from IANA or CF) about how they configured CF request rejection and how one should love CF's flat rate static serving be appropriate?
Am I missing something?
It's quite sad to see incompetence at these levels. This is a static page that loads JS. They even inject styles via JS...
It’s not incompetence they are intentionally doing as much as possible via javascript to reduce their bandwidth costs. This is because most access to the domain is automated (not using a browser, using curl etc) and doesn’t load javascript
They use Cloudflare, which does not charge for bandwidth.
For free users. But cloudflare picks and chooses its free users.