by Rik Farrow
The Slapper worm appeared sometime after midnight on January 25. Within minutes, Slapper, also known as Sapphire, had infected tens of thousands of vulnerable systems attached to the Internet. The pace of infection, and the flooding caused as the worm spread itself, were stunning. But within about 14 hours, traffic on the Internet had mostly returned to normal.
As yet another worm that feeds on unpatched Microsoft servers, Slapper is not extraordinary. What makes Slapper interesting is both its rate of spread and that it quickly managed to penetrate so many organizations that should have been secured. Bank ATMs stopped working and financial organizations sent people home two days after the attack because their networks were still infested, all because of a very simple worm.
The Slapper worm shows us several things. One, that systems are still not being patched, although sysadmins could claim some valid excuses. But more importantly, that there are many ways that hostile code, and presumably attackers as well, can penetrate firewall-protected networks at will.
The Slapper Worm set speed records. The number of infections doubled every 8.5 seconds. 90% of all vulnerable systems had been infected within 10 minutes. At its peak, the worm sent out packets at a total rate of 55 million packets per second. What made Slapper such a speed demon?
For one thing, Slapper is small. Only 376 bytes of code, 404 bytes with IP and UDP headers, the attack easily fits into a single packet. By comparison, CodeRed used 4K, and the much more complex Nimda 60K. And Slapper used UDP instead of TCP. While CodeRed could only scan perhaps ten systems per second, Slapper was limited only by the available network bandwidth. CodeRed sets up a TCP connection, which requires waiting for a response from the potential victim, that the target port is either open or closed. Slapper simply sends its single packet payload as fast as the infected server can write to the network. Slapper actually spread hundreds of times faster than CodeRed.
Yet, within 14 hours, traffic on the Internet had pretty much returned to normal, unlike the attacks of CodeRed and Nimda that remained heavy for weeks. With Slapper, it was not so much that sysadmins responded by stopping and patching SQL Server (on Saturday morning). It was that Slapper was such an aggressively scanning worm that ISPs took to blocking it in self-defense. ISPs at the edges of the Internet took to blocking port 1434/UDP to prevent their customers' networks from flooding them. And many ISPs had to deal with colocated Windows 2000 servers that were sharing their internal networks--and flooding those network. They either filtered, shutdown or disconnected infected servers until they could be patched.
Like the other recent worms, patches for the vulnerability used by Slapper had existed for a long time before the attack. Microsoft had posted a security bulletin to fix the vulnerability used by Slapper, MS-02-39, back in July 2002. David Litchfield, a well-known security researcher in England, had uncovered two issues with MS SQL Server, and posted his own advisory after Microsoft announced a patch (see Resources). One issue involved a denial of service, but the more serious problem included a buffer overflow of the Microsoft SQL Server through the monitor port, 1434/UDP. While scans for port 1433/TCP have been common for the previous year, with attackers searching for MS SQL servers vulnerable to attacks against mis-configured or poorly secured servers, the vulnerability that appears via 1434/UDP raised new issues. The service listening to this port, ssnetlib.dll, runs as Local System, and anyone exploiting this service would have complete control over the targeted system.
Litchfield even posted an exploit for this vulnerability, which he claimed to have done so that administrators could understand how attackers might use the vulnerability to attack their systems. The Slapper worm's author borrowed some ideas from Litchfield's exploit example, although Litchfield had not written a worm.
MS-02-39 was not the only patch for MS SQL Server that fixed the vulnerability. Several other patches, include SP3 for MS SQL Server, and MS-02-061, also included newer, patched versions of ssnetlib.dll. But in October, Microsoft released a hotfix for MS SQL Server that was supposed to fix a performance issue, Q317748. This patch had been created before the security issue had arisen, but not been made generally available. Q317748 regressed ssnetlib.dll to an unpatched level, vulnerable to Slapper.
In theory, no one should have been confused by Q317748. MS SQL Server patches must be unpacked, then "hand-installed" by an administrator who copies the new files into place. Administrators should have noticed they were copying an older version over a newer version of ssnetlib.dll. But theory is not the same as practice. And even Microsoft's internal network was still plagued with Slapper by Monday morning, two days after Microsoft's IT department had been issuing instructions for stopping and updating SQL Server. And that implies that Microsoft's own, internal systems had not been patched properly either. Microsoft can hardly go about blaming administrators for not patching systems if they cannot practice what they preach.
Another problem for many administrators is that they could easily not have know they were running a variant of MS SQL Server. MSDE, Microsoft Database Engine, is an embedded version of MS SQL Server. Many software packages include MSDE, and these were vulnerable as well (for a list of Microsoft applications that embed MSDE, (see Resources). Included among third party applications using MSDE were several security products, including McAfee Centralized Virus Admin, Trend Micro Damage Cleanup Server, and even both of ISS' flagship security products, RealSecure 7.0 and Internet Scanner. Although ISS received credit for producing an advisory about the worm, naming it Slammer, they never admitted to having shipped the vulnerable component, MSDE, long after Microsoft had issued a patch for it. In other words, if a company used ISS Internet Scanner to search for vulnerable versions of MSDE, they would have found that the system with ISS installed was itself vulnerable-- and not known that the vulnerability came from ISS.
It wasn't just Microsoft that had problems with Slapper on its internal networks. Bank of America Corporation, one of the largest US banks, said that many of its customers could not withdraw money from its ATM network of 13,000 terminals. Customers of Imperial Bank of Commerce in Toronto also were unable to withdraw money using ATM machines for part of Saturday. American Exppress, A.C. Nielsen, PriceWaterhouseCoopers, and Citicorp all were hit hard as well. Many businesses sent employees home Monday morning, as their networks were still infested with Slapper.
What most analyses of the Slapper ignore is how did an exploit, that uses an unusual UDP port, get into these organizations in the first place? Well-configured firewalls include a default deny rule, and only permit a very limited number of services to enter the internal network. UDP-based services, with the exception of DNS and some media-streaming applications, are generally considered very insecure and dangerous. Allowing any, arbitrary, incoming traffic through a firewall is unusual, as most incoming traffic gets directed to a handful of (hopefully) secure and patched services, such as mail and Web servers.
Of course, CodeRed and Nimda also got into internal networks too. CodeRed seems a bit more likely to get inside than Slapper, as it relied on HTTP, a service that does get passed through firewalls. Nimble Nimda easily got inside, because it not only used HTTP, it also include email vectors that relied on buggy versions of Internet Explorer. Once inside an organization, Nimda would scan aggressively on the local network for vulnerable Web servers. So, no mystery about how these worms got inside.
But Slapper uses a port only relied on by clients of MS SQL Servers. MS SQL Server permits clients to use two different mechanisms for sending queries: pipe requests over ports 139 and 445/TCP, and direct requests to port 1433/TCP. By sending a UDP packet to port 1434/UDP containing just the value "4", the client should get a response from the MS SQL Server telling the client which method to use for requests.
While all of these ports might be left open on a pirated install of Win2K Server on a home network, it is hard to imagine that any of these ports would be left open on servers installed behind firewalls. Also, scanning of ports 139, 445, and 1433/TCP has been very common, and any person managing a firewall or IDS could hardly not have been aware of the activity. To consider that a relatively unknown UDP port would be left unprotected is amazing. And indeed, most firewalls did block packets addressed to port 1434/UDP.
Instead of entering through firewalls, like CodeRed and Nimda, Slapper most likely entered internal networks through other means. Home systems attached to the Internet would certainly have become infected if not protected by firewalls. And when that home system connected to an organization through a VPN tunnel, Slapper would soon have infected the internal network. VPN's connecting different locations could also have passed Slapper. It was rumoured that the failure of bank ATMs was caused by overloaded VPN tunnels.
All it took for Slapper to get into an organization was one unprotected connection to a system on an infected network. Worms like Slapper argue for filtering traffic that arrives on VPNs, instead of trusting all traffic from the remote end of a VPN tunnel. Logically, this makes sense, as the organization at the remote end of a VPN tunnel may have different security policies and implementations.
Slapper infections also suggest that more firewalls are needed. Any telecommuter working from home over a VPN must have a properly configured firewall installed (see NetDef June 2002). You should consider installing firewalls within your organization, partitioning it, so that an infection in one part does not affect all parts of your organization. Internal firewalls need to be fast, and configured to permit only traffic to expected services, with valid source addresses. Configuring such a firewall is not that difficult, and silicon-based firewalls, like Netscreen and Watchguard's Vclass can support high bandwith connections to departments to be isolated. Highend routers could perform this filtering as well.
If your organization already has internal firewalls, you must have a method to centrally manage those firewalls, so that you can further isolate sections of your organizations as needed. With Slapper, simply blocking 1434/UDP would have stopped the infection, although you might still have servers behind firewalls that are infected, the infection could not spread to other firewall-protected regions.
The Slapper worm has many lessons for those in IT security. First, that patching has not been complete or effective. Six months would seem to be more than enough time to install a critical patch. MS-02-039 was released by Microsoft on July 24, 2002, and rated "critical" by Microsoft. Of course, lack of awareness of just how many systems included MSDE might have prevented administrators from knowing which systems they needed to patch. But systems like HFNetChk (www.shavlik.com) easily identify software that has been installed and that needs patching. And vendors who embedded software with a critical vulnerability should also have announced the issue to their customers, and offered to provide the patch.
While poor patching practices lead to most of the Slapper infections, improper firewall configuration, and lack of filtering at the end of VPN tunnels, allowed Slapper to penetrate even supposedly secure organizations, like Microsoft and Bank of America.
And, when all is said and done, it appears that Microsoft's Trustworthy Computing initiative still has light years to go before it can claim any success.
The initial advisory from Next Generation Software: http://www.nextgenss.com/advisories/mssql-udp.txt
The latest MS patch that fixes the bugs in SQL Server 2000 and MSDE: http://www.microsoft.com/technet/treeview/default.asp?url=/technet/security/bulletin/MS02-061.asp (First bulletin was MS-02-39)
MS SQL SP3: http://www.microsoft.com/sql/downloads/2000/sp3.asp
A list of Microsoft products that contain MSDE 2000: http://www.microsoft.com/technet/security/MSDEapps.asp
Analysis of the spread of Slapper (aka Sapphire) by security researchers: http://www.caida.org/outreach/papers/2003/sapphire/sapphire.html
Analysis of the code used by Slapper:
http://www.eeye.com/html/Research/Flash/sapphire.txt