One question the article doesn't answer is: why are they cacheing at all? If your cache is that big it isn't a cache. How much bigger is the dataset in question? There are 250 billion entries. Assuming 80/20, that implies 1.25 trillion records?
What's the speed of service/response time relative to the data source?
At that point it might be enough to replace your multiple caches with fewer in-RAM databases?
It's an interesting problem.
Maybe I'm misunderstanding, but this powers 1.1.1.1, it doesn't front an internal dataset. A cache miss hits a nameserver. Which is to say, the dataset is "every DNS record in the world"
I think the question is probably more along the lines of - why not do a database with 100 TB of storage/records instead of a cache? tomato / tomato.. especially with smart caching in front of database. 100TB of flash is a good bit cheaper than 100TB of memory
Because it would be slower and have different scaling requirements than the ones they want.
It's not 100TB of data. It's probably 50 GB of data on each of 2000 servers. Because it's a cache. What is the point of a central cache if it's as slow to access as the original data?
TFA gives numbers closer to 5GB.
I'm no expert but presumably all of throughout, latency, and churn. DNS is approximately a giant KV store where the typical record has a TTL of ~5 minutes.
This is smart, task-specific caching in front of database.
You have to cache, cloudflare doesn't know all the records ahead of time, they have to do recursive lookups to the authoritative servers that own the records and that is only good for the period of the TTL of the record. There is no "global" DNS record database or something like that.
>that is only good for the period of the TTL of the record.
Not really, TTLs are often short, but IPs might not change for years.
You can probably generate your own TTL, at scale, and avoid many DNS requests.
Why would anyone want to use a DNS resolver that tampered with records on a large scale? The TTL is intentionally set by the originator of the record.
Or alternatively, if you don't tamper why would I want to use a service that serves stale data?
Every distributed system serves stale data.
You can define away ‘stale’ by picking a consistency model, but look inside the consistency machinery and you will see fresher data you aren’t allowed to have yet.
That's like refusing to use a fridge, insisting on letting your produce gradually spoil on the counter because "all harvested produce is in a state of decay anyway".
In DNS, the owner of each record has full control over its TTL. Intermediary DNS servers are required to honor them and are not permitted to replace TTLs with their own.
DNS servers do in fact do that but it would not be a good look for the world's largest DNS provider.
I don't think cloudflare cares about how it looks, also I think Google is bigger.
Actually that is not true. The IETF has expanded the definition of “TTL” and explicitly permits resolvers to serve “stale” RRs beyond their expiration time.
https://www.rfc-editor.org/info/rfc8767/
As a corollary, there is obviously no floor on refetching unexpired RRs, of course, except for efficiency concerns.
That's only when the authoritative server cant be reached though
You are obliged to pass on the TTL, you're not obliged to cache according to it.
At least in my country (UK) I know of no law relating to DNS caching.
Why throwaway perfectly good data every few minutes that is only modified every couple of years, just so someone can move their domain quickly when they eventually wish to? It is my contention that a [caching] DNS service can do far better. Trusting user (domain owner) input blindly is not for me.
It's not some sort of public law with public enforcement, but it is in the RFCs that govern the protocol.
I should be a bit clearer here; the TTL is an upper bound on how long it can be cached. Caches are free to consult more frequently but not less frequently. That said, out of respect for upstream cache operators and authoritative servers, most DNS caches honor TTLs as best they can.
The IETF isn't the internet police. You don't have to follow its advice.
No, but if you didn’t, the internet wouldn’t really work all that well. It was the fact that participants, despite being independent, all agreed (either explicitly or implicitly) to adhere to the standards that it became a global network. If they hadn’t, the result would have been more of the same: independent networks that only had narrow interoperability at best. It’s a textbook example of how global cooperation can yield incredible results.
Advocating to do things against agreed-upon standards without a compelling reason and without giving due consideration to the adverse consequences is one of the hallmarks of a bad engineer. Even Microsoft played nice with Internet standards for the most part (although with some notable exceptions at the application layer that got them well-deserved criticism).
It was the fact that adhering to the standard was in the best interest of each participant. When it isn't, they don't.
It's impossible to know how often participants chose to stick to the standards anyway despite it not being in their best interest. Do you have any specific examples?
Linux routers responding ICMP from 192.0.0.8 when not having an IP address (currently being standardized). Clouds using class E address space (240.0.0.0-255.255.255.254). Routers that drop IP packets with options. The entire concept of NAT until the existing behavior was eventually standardized. Unnumbered interfaces.
then they would be breaking DNS at scale.
It's a recursive resolver. The global DNS dataset is not something you could collect to serve directly vs caching from observations.
The data source is authoritative name servers operated by third parties, some of which are slow on their own, some of which are behind slow or lossy networks. Origin response times vary between probably 1 ms and 2 seconds +/- origins that never respond.
They’re adding the cache consumed across all of their servers. It’s not one giant deep cache.
The simple answer is that if you didn't cache, DNS traffic would skyrocket, and the load would pile up on the authoritative servers, which were intended to be small, and during the early days of the Internet, were frequently on bandwidth-constrained links.
DNS is designed to distribute query load to the edge as much as possible, and that's enabled by caching. It just so happens that "the edge" is now becoming concentrated among a small set of providers because they wanted to make a business out of it.[1] They knew that this would be expensive going in, though.
[1] Nobody has to use 8.8.8.8 or 1.1.1.1. Most people can use their ISP's cache or a local cache instead without any noticeable difference in behavior.
The problem is there is a noticable difference in behavior because the ISP cache is overloaded so queries take longer. Sure, that's not everyone's experience, but there's a reason people chose to use alternate servers.
> If your cache is that big it isn't a cache.
This is an incorrect statement. Caches do not have a requirement of being smaller than their source data set. CDN is an example of a cache that generally matches the size of the source data.