Is there a practical way to enumerate all the registered internet domains? EG by asking DNS servers for all the domains they know about, and repeating over all DNS servers they know about?
EDIT: apparently, "asking DNS servers for all the domains they know about" is not something you can really do anymore for security reasons. Guess that idea won't fly lol
Enumeration of the entire DNS space is not available in general, but it does appear that some TLDs offer complete zone files for legitimate research purposes, see for example https://czds.icann.org/help#zone-files
Check out the "Root Zone Database",
https://www.iana.org/domains/root/db
(this doesn't have .yu)
This https://www.whoisxmlapi.com/whois-database-download.php is a start, but there's new ones every second so you're going to need to update a lot.
> apparently, "asking DNS servers for all the domains they know about" is not something you can really do anymore for security reasons. Guess that idea won't fly lol
There are actually a few nameservers that will just give all their domains to anyone who asks [0], but they are very much in the minority.
[0]: https://github.com/acidvegas/mdaxfr#tlds-that-allow-axfr
You could probably extract a lot from https://commoncrawl.org/
Here's the latest (although, it looks truncated at those having > 1M pages),
https://commoncrawl.github.io/cc-crawl-statistics/plots/tld/...
The complete list hides in the web graph:
https://data.commoncrawl.org/projects/hyperlinkgraph/cc-main...
and the specific file that's every host we've seen in the latest 3 crawls is:
https://data.commoncrawl.org/projects/hyperlinkgraph/cc-main...