About a month ago, I became unable to access e-mail, and I was also unable to log in to some of my servers. The problem persisted for an hour or so and then cleared up. A few days later, the problem reoccurred. Since the servers are located remotely, and since I was unable to log in to the servers during the time of the outage, it was not possible to see what was going on in real time. I looked through various mail server logs and system logs but could not find anything that looked particularly out of the ordinary.
Finally, one day, the problem persisted for several hours. After several attempts, I was able to remotely log in to one of my routers, and after enabling the system monitor (on Cisco routers by issuing the term mon command), I started to see repeated buffer overflow messages. This was my first clue that something was wrong since a buffer overflow message from a router indicates that the router is having serious issues managing the amount of traffic present on its interfaces.
Next, I issued a show process cpu history command to the router to view a graph of router CPU load. I immediately saw that, for the past several hours, my router had been operating near or at 100-percent capacity. I used the show if fa0/1 command to review statistics for the WAN facing interface on my router and noted that I had more than 7 million dropped packets in the last week. For some reason, someone on the Internet was directing a huge amount of traffic my way — so much that my router had failed to deliver 7 million packets. I do not know how many packets were successfully handled by my router during the last week, but that number is probably hundreds of millions.
My first indication of what was going on came when I looked at where this traffic was directed — all at my Domain Name Server (DNS). What followed was a crash course in something called a DNS amplification attack — just one of the many weapons used in an increasingly hostile Internet environment.
Figure 1. Any computer can ask a DNS server to find the IP address for a particular domain name. An answer is given if the server is authoritative for that domain.
Domain Name Servers are a critical part of the Internet. Humans use domain names to access servers on the Internet, but browsers and other applications use IP addresses. As Figure 1 shows, DNS servers resolve a domain name to an IP address. Any computer on the Internet may ask a DNS server to find the IP address for a particular domain name — broadcastengineering.com, for example. If the DNS server is authoritative for that domain, meaning that it is the original source of information about that domain name, then it will give the answer. However, a properly configured DNS server will not return an answer if it is not authoritative. If you ask the DNS server at broadcastengineering.com for the IP address of google.com, the Broadcast Engineering DNS server should not give an answer.
Some servers known as Recursive DNS servers will return an answer to a query even if they are not authoritative for the domain in question. They do this by making a quick look-up for the domain on another DNS server and then delivering the response back to the requestor. Unfortunately, I had misconfigured my DNS server to be recursive — a major mistake. (More on that later.)
If a user tries to view a website and the name server is slow to respond, then the end user perceives that the website is slow, even if the web server is functioning perfectly normally. For this reason, DNS severs tend to be fast servers connected to large pipes.
So, the question was: Why were bad guys out there directing a lot of traffic at my DNS server? If an attacker can figure out how to efficiently use DNS servers to direct a lot of traffic at a target, then they have an effective weapon in the cyber war. This is exactly what a DNS amplification attack does. By making a small query, an attacker can get a DNS server to respond with a large reply. This is called an amplification attack. To give you an idea of the amount of amplification that is possible, a 60B Extended DNS (EDNS) query can yield a response of more than 4000B — an amplification factor of 66.
But, if an attacker makes a small query and the DNS server responds to him or her with a 66X reply, that is not very useful. The final part of the attack is that the attacker spoofs the source address so that it looks like it is the victim. Now that all the pieces are in place, let’s look at how the attack using my server worked.
Figure 2. An attacker pretends to be a victim asking for a response. The DNS server then floods the real victim with a huge, unsolicited reply.
As Figure 2 shows, an attacker sends an EDNS query to my server, but he or she spoofs the source address of the message so that it looks as if the query is originating from the victim. My server dutifully responds to the query, but instead of sending the reply to the attacker, my server sends it to the victim since my server thinks that it is the victim who is asking for the information. My DNS response shows up at the victim’s IP address unsolicited. That really is all there is to it. Act like you are someone else, ask a small question, and watch them receive a huge reply.
However, here is the rub: It is not a single attacker. My server was hit with requests from many different attackers. Someone operating a bot-net had instructed the computers to all attack the same victim. A bot-net is a federation of compromised computers, all commanded to execute commands remotely by people who control the bot-net. Bot-nets are comprised of many thousands of laptops, desktop systems and even computers at businesses. The one thing these computers have in common is that, at some point, people have executed programs on these computers that compromised their security, allowing the computer to be remotely controlled without the owner’s knowledge. Analysis of my DNS logs showed that hundreds of computers were being used to generate thousands of similar requests per minute, all directed at the same target.
What can a media company do to guard against having its name servers coopted into participating in a DNS amplification attack? The first thing is, if you operate your own DNS servers, ensure that your servers are not recursive, meaning that they will only answer for the domains you control. Second, configure the DNS server’s firewall using the RATE command to ensure that if multiple requests come in from the same IP address in short order, then those IPs are automatically black-listed. Third, if you are using a Cisco router to connect to the Internet, consider enabling Unicast Reverse Source Path Forwarding or Unicast RPF. RPF limits attacks by verifying that the requestor is not spoofing the IP address.
I realize that there may not be even one reader who is operating his or her own DNS servers on the Internet. However, I expect that there are a number of media companies operating Web servers, e-mail servers and other systems that rely on Internet connectivity. The DNS amplification attack is only one tool in the attacker’s tool box. Attacks exist for every kind of server. So, my hope is not that you will secure your DNS servers to keep from being an unwitting agent in an attack on a third party. It is to tell you about my real-world experience, to let you know that attacks are happening and to tell you to be vigilant.
—Brad Gilmer is president of Gilmer & Associates, executive director of the Video Services Forum and executive director of the Advanced Media Workflow Association.
