by Rik Farrow
Monday, October 21, 2002 was not the day the Internet stopped. That evening, the 13 DNS root servers suffered a heavy denial of service (DoS) attack that failed to disrupt service. The attackers shifted their focus to another set of critical DNS servers before ending the attack.
Even though the attack failed, it did serve to attract a lot of press attention,
with the usual mentions of possible cyberterrorism. But no individual or group
stepped forward to
claim responsibility for the attack, which not only claimed no lives nor injured
anyone, but also failed to disrupt use of the Internet.
The Domain Name Service is critical to the proper operation of the Internet. In "DNS Troubles," [August 2000, http://www.spirit.com/Network/net0600.html], I detailed a number of issues surrounding DNS. In this article, I'll focus on the top of the DNS hierarchy (the root servers), and the even more important top-level domain (TLD) servers.
While an attack that prevents access to TLD servers is possible, it would most likely be of relatively short duration, and would not have catastrophic consequences.
In addition, there are steps you can take to insulate your critical business functions from suffering side-effects--unless you are in e-business. For e-businesses that rely on a steady stream of customers, losing DNS for any amount of time will result in loss of potential customers, at least for the duration of the attack.
The beauty of DNS is that it is a hierarchy of servers, allowing organizations
to manage their own online names using their own DNS servers. But DNS functions
in a top-down fashion, with
requests starting at the root of the inverted tree of DNS servers: the root
servers.
The thirteen root servers form the top of the tree, with names ranging from A to
M, and all in the domain root-servers.net. Most of the root servers are located
in the US, although there are
also root servers in Sweden, London and Tokyo. Because DNS lookups start with
the root servers, your DNS can't go looking for, say, A.ROOT-SERVERS.NET, but
instead uses a hard-coded list for these servers (called a cache or hints file).
For example, imagine you just felt the building you are in shake, and are
wondering what has happened. You open up a browser window, and enter: cnn.com.
Within a second or so, the CNN
home page begins loading. During that brief pause (ignoring the issue of DNS
caching for now), your system sends out a DNS request to your local DNS server.
Your local server first visited one of the root servers, and asked for the
address of a server for .com. The root server
responds with the address for Top Level Domain (TLD) server, and your local DNS
server sends a request for cnn.com. Finally, your server sends a request to
cnn.com, and gets back an IP address that your Web browser can use to start
loading the home page for CNN.com, say, 64.236.16.116.
Until about mid-2000, the root servers also handled the generic top level domains, or gTLDs. The gTLDs are the familiar .com, .net, .org, .gov, .mil, and .edu, groupings that were created when the Internet was a US-only network. Later, country-code TLDs (ccTLDs) were added, such as .ch, .gb, and .jp. The root servers, which sometimes were just UNIX workstations sitting in someone's office or lab, had come under increasing loads, and the management and availability of these servers had also become more critical.
In June of 2000, RFC 2780 (see Resources) spelled out best current practices for root servers. Best practices include the capability of each root server to handle three times its measured peak load--providing failover capability for the other servers if the need should arise. Part of this failover capability includes not just server performance but also enough network bandwidth to handle any additional loads.
The eight US gTLDs are all currently operated by Network Solutions, and run on
IBM AIX servers using the same software. As with root servers, there are also
international gTLDs, in Hong Kong, Tokyo, Stockholm, and two in the UK. The
gTLDs get many more requests
than the root servers, and are, in fact, more critical to the operation of DNS.
The root servers simply point to the gTLDs and ccTLDs. These servers, in turn,
return the addresses of the
authoritative names servers for most domains.
Earlier, we detailed a scenario for looking up an address for CNN.com while
ignoring the implications of caching. DNS servers use caching to retain answers
provided through previous lookups. In the CNN.com example, the local DNS server
might have recently cached the information about the address for CNN.com, the
name server for CNN.com, and the .com
gTLD servers. Every answer received by a DNS server comes with a timeout value,
the time to live (TTL). When the TTL expires, the DNS server must get a fresh
answer.
The gTLD servers have TTLs of 172,800 seconds--48 hours for those of us who
don't
reckon time in seconds. CNN.com, on the other hand, actually sent eight address
records, each with a timeout of five minutes. Anyone else using the same local
DNS server would get the cached answer for CNN.com for the next five minutes.
Later requests might produce the same list of addresses, a rotated list of
addresses (so that a different server becomes the first choice), or a different
set of addresses. For CNN's system administrators, this means they can change
addresses and expect the change to take effect in no more than five minutes.
Checking the TTLs for the .com gTLD, the answer was two days. That is, a local DNS server would only have to visit a root server once every two days to get the current list of .com gTLD servers. A little further checking revealed that all of the gTLD servers that I checked had TTLs of two days. Of course, if a local DNS server's cache was flushed, or the server was restarted, it would start over with an empty cache, and require visits to the root servers to get the addresses for the gTLDs and ccTLDs.
The DNS server cache reduces workload for any frequently visited DNS servers, as
well as reducing its own workload, and the amount of traffic on the Internet.
But the use of caching
has implications for attacks on root and TLD servers as well.
According to posts made by Paul Vixie, the operator of the F root server, the
October 21th
DoS attack lasted for almost an hour, and consisted of ICMP Echo Request
packets--ping packets, in other words. Vixie stated that he worked with his ISP
to "push back" the attack. I am assuming that Vixie contacted his ISP and asked
that ICMP packets either be rate-limited or dropped until the attack ceased. At
that point, the attacking packets stopped reaching the F server, although the
packets were still affecting the ISP's network. I wrote about these techniques
for reducing the impact of DoS attacks in "DDoS is Neither Dead nor Forgotten,"
[February 2001, http://www.spirit.com/Network/net1200.html].
Suppose that the attacks against all the root servers had continued all day, and
actually managed to prevent most requests from reaching the root servers.
During the attacks, some servers were
actually answering less than 50% of the requests sent to them (see the CAIDA
graphing tool in the Resources section), so it is not as if the October 21
attack had no impact at all. In
an attack that involved a much heavier volume of packets, the root server
request rate might fall to the point where few requests, say less than 10%, were
being answered. For those systems making requests, it might appear that the
Internet has stopped working.
But for those using DNS servers with caching, information about finding the TLD
servers will remain valid in their caches for two days! Suddenly, the root
servers appear a bit less important,
and the TLD servers more important. And the answers provided by TLD servers get
updated every twelve hours, so these answers will timeout sooner, but not
immediately. This illustrates
the wisdom behind having multiple local DNS servers, as the caches used by these
servers become an important resource during any large attacks against the root
or TLD servers.
BACKUP PLANS
For e-businesses, DNS and the root and TLD server remain a critical resource.
Without a means for finding the address that goes with a domain name, no one can
visit a particular site.
The owners of these sites can only hope that the answer has been cached--and
that the TTL for the address has not yet timed out.
Downloading entire TLDs is not a good plan, either. The .com zone file was over 2 gigabytes in size at the beginning of 2002. And that is just one TLD. It takes the TLD servers almost a half hour just to load the .com zone file, and these are servers capable of answering thousands of requests each second. The F root servers handle an average of nearly 2000 requests per second each, with much higher peak loads.
If your organization relies on frequent Internet communications with a number of other organizations, you might consider creating a list of those organizations' DNS servers' names and IP addresses. Even though your DNS server will not be authoritative for those domains, in an emergency, your local DNS could be primed with entries for those Internet addresses that you just cannot survive without.
A cleaner way of accomplishing this would be to become a secondary DNS server
for domains
that represent important partners. Becoming a secondary DNS server means that
your local server will maintain an up-to-date copy of the remote domain's
information without
requiring any addition work or maintenance. You, and the partner site, just
need to adjust your DNS server configuration so that you can perform zone
transfers.
Having secondary (backup) DNS servers is an old Internet tradition. Another
part of this tradition includes having alternate mail server addresses (MX
records in DNS), so that
if your email server became unavailable, your email would be stored at the
secondary site until it could be forwarded. You're probably aware of the
security considerations of having
some other organization receiving and storing your email-- but then you are
already encrypting your sensitive emails, aren't you?
The Internet functions as well as it does through flexibility and redundancy.
Having 13 root servers and 13 TLD servers increases the reliability of the
Internet. Even if all 13
servers became unreachable, many people wouldn't even notice any change in
surfing, as local servers will have the necessary answers in their caches. At
the same time, keeping
root and TLD server up and available is not just business, but a personal
challenge for the people who run the servers, and the networks they connect
with. I expect these folks will do everything in their power to keep the root
and TLD servers available.
For your own organization's sake, consider setting up secondary servers. There
are even sites that specialize in finding partners. Then you too can play a
part in keeping the Internet running
smoothly.
Internet Software Consortium's page on the F root server: http://www.isc.org/services/public/F-root-server.html
RFC 2870 on Best Current Practice for root servers: http://www.isi.edu/in-notes/rfc2870.txt
CAIDA page providing access to chart of root server response times: http://www.caida.org/cgi-bin/dns_perf/main.pl
David Moore's USENIX Security paper about detecting DDoS attacks: http://www.caida.org/outreach/presentations/usenix0108/dos/
ICANN Targets DDoS Attacks, mentions possible Federal action to "encourage" adoption of source address verification: http://www.eweek.com/article2/0,3959,651686,00.asp
NANOG thread index (for time of Oct 21 attack): http://www.merit.edu/mail.archives/nanog/2002-10/
Great article by Joshua Green debunking the "cyberterrorist" threat: http://www.washingtonmonthly.com/features/2001/0211.green.html