Frankly, I don't understand why someone would even try to crawl Hacker News.

There is an official dump which doesn't even require parsing HTML at all: https://console.cloud.google.com/marketplace/details/y-combi...