by Rik Farrow
Patching vulnerabilities does more to help ensure network security than any other security-related practice. Unfortunately, however, most organizations and users don't consistently patch their systems.
According to the security web site Attrition (www.attrition.org), 99 percent of the 5,823 defacements of Web servers recorded during the year 2000 could have been prevented if patches had been installed on the affected systems.
And things aren't getting any better. Anti-virus software vendors reported that SirCam and Nimda were the fourth and fifth most commonly encountered viruses (respectively) in September 2002. Both viruses rely on old, unpatched versions of Internet Explorer to function.
Then there's CodeRed v2, which is still attacking systems more than a year after it was first launched.
If patching were easy, organizations and users would do it consistently.
Instead, system administrators and users generally avoid patching their
systems, for a variety of reasons. I believe the two major reasons are lack of
resources to devote to patching, and a
resistance to patching systems that are currently working.
The task of patching takes both determination and the will to expend the resources necessary to make it happen. One large site, the San Diego Supercomputing Center (SDSC), shows that not only is patching doable, but it also makes an enormous difference in network security.
The last time I wrote about patching (see "Patches Make Perfect," April 2000, page 88) I didn't know about SDSC. Located on the San Diego campus of the University of California, SDSC is a research organization with enormous amounts of disk space and computing resources. Like many other research sites and universities, SDSC has no firewall. Routers at SDSC do perform the type of routine filtering--for example, blocking packets with spoofed source addresses--that any router should be configured to perform. But according to SDSC, it's had no root exploits in more than two years. The center attributes this achievement to two practices: It has phased out all network applications that use plaintext passwords for authentication, and the staff routinely patches the network's systems.
SDSC's network includes a mix of different flavors of UNIX systems, and an almost equivalent number of Windows desktops, for a total of nearly 800 systems. The techniques used to patch the two operating system types are different, but they do have some similarities.
Many research papers have been published about patching (see www.usenix.org/publications/library/proceedings/), and almost every one starts the same way, citing the three things that must be done to begin a site-wide patching regime. First, the organization must create a policy that demands patching. Without administrative support, the owners of individual systems can resist patching their systems. At SDSC, the policy supports patching-- if someone doesn't want to allow SDSC admins to patch the systems they own, they cannot attach to the internal network. Keep in mind that, since there's no firewall, "internal" is a relative term in this case. However, certain resources become unavailable to those who wish to remain "outside."
The second step in initiating site-wide patching is to perform a hardware and software inventory. You need to know exactly what types of systems are connected to your network, including hardware architecture and operating systems. This inventory work can be automated using commercial scanning products or free software, such as Nessus (nessus.org).
No matter how you create your inventory, you should build a database of system information that includes the architecture, the operating system, and all network services discovered. Scanning tools will also discover potential vulnerabilities in network services, but finding these is really secondary to patching. If the systems were patched, there would be no vulnerabilities to find. Figuratively speaking, vulnerability scanners tell you that your roof is leaking when you already know that your roof needed patching.
The third practice that most researchers agree should be performed is bringing all systems up to the same patch level. Your organization might be running several versions of Windows or Solaris. The goal is to get each system up to the same patch level. At SDSC, this process includes phasing out support for old operating systems and phasing in new operating systems versions.
In each case, reference systems are built from distribution CDs, standard applications are loaded, and then patches are installed. New systems are built using the reference systems as the source. In the Windows world, cloning tools such as Ghost are used. In the UNIX world, procedures vary depending on the operating systems' requirements. When installing Linux systems, cloning of hard disks is simple and expeditious. For larger systems with more complex hardware, the vendor's installation software, such as the Solaris JumpStart automated install, can be used.
Once all systems have been brought up to the same current patch level, a lot of the work in the overall process has been completed. While you'll still need up-to-date anti-virus software on your Windows desktops, you'll no longer be affected by many of the email viruses in circulation today. Your UNIX systems will be safe from the most common attacks used by automated attack tools and script kiddies. You won't have won the war against attackers, but you'll certainly have gained a reprieve.
The next step in the patching process is to maintain your systems at the most current patch level, which is much easier said than done. In September 2002 alone, Microsoft issued four security bulletins, affecting potentially every Windows desktop in many organizations. But does this mean that every time Microsoft sneezes, you have to jump?
Not necessarily. A report on patch handling published by NIST (see Resources) suggests a form of triage to be used when new patches are announced. Rather than becoming a slave to patch announcements, the NIST paper suggests (as do others) that you organize a Patching and Vulnerability Group (PVG). The PVG meets routinely to discuss and assess new vulnerabilities and patches after they've been announced. Only when an emergency situation arises would the PVG meet more often than, say, once a week.
Occasionally, a vulnerability will appear to be so dangerous and easy to exploit that it must be patched immediately. More commonly, patches can be installed in a slightly more leisurely manner, and then subsequently tested.
At SDSC, the administrative group, along with volunteers among the workstation users, tests new patches. Rather than push out a patch that might break some critical application, the group tests new patches for at least a day (and sometimes several days) before releasing them. However, the systems being used for testing must run similar applications to the systems on which the patches will be installed. Because the activities of system administrators and users are actually very different (outside of email and Web browsing), this is not as easy as it sounds. The testers also must verify that the patch actually fixes the vulnerability, and that it doesn't re-open other vulnerabilities (a problem that's arisen in the past). This is a situation in which a vulnerability scanner or a Open Source tool like Nessus can really prove its worth.
Once the patch or patches have been tested and proven to be safe and effective, the patches can be installed. Organizations that use Windows Domain authentication, and that have brought all systems up to the same patch level, can use Microsoft's System Management Service (SMS). The Windows administrator uses the SMS installer to create snapshots of the reference system before and after installation of a patch or patches. Then, the SMS installer compares the snapshots and creates a package describing the differences between the unpatched and patched systems. This approach minimizes the amount of files transferred, as well as capturing other subtle changes included in the package, such as alterations to registry keys and access control lists (ACLs).
SMS is designed so that each time a user logs into the domain, his or her workstation is checked to determine whether the SMS client is installed and running. If the client has not been installed, a NetLogon script installs the client. SMS can either advertise patches, so that systems can "pull" those patches, or it can "push" patches. As patching Windows systems often involves rebooting, many sites prefer to set up SMS so patching occurs overnight.
At SDSC, the Windows IT group has taken additional steps to help ensure that SMS patching goes smoothly. The BIOSs' of new systems are set to boot from disk first, so that the system won't hang during a reboot if someone leaves a floppy disk inserted in a machine. The IT group also has removed the Shutdown option from the Windows NT, 2000, and XP Secure Attention Sequence dialog boxes, so that users are not tempted to shut down their systems at night. Because patching, and possibly rebooting, happens at night, the chances that someone will lose edits to an important file due to a mysterious system reboot caused by SMS are dramatically reduced.
For UNIX, SDSC uses a different tool, cfengine (Configuration Engine). cfengine was developed by Mark Burgess, and can be found at www.cfengine.org. This tool can be used in a client-only approach, or in a client-server approach. In either mode, cfengine follows configuration files that perform routine administration tasks on the UNIX systems running them, such as removing old temporary files, checking ownership and permissions, and fixing symbolic links. But cfengine can also be used to install patches.
System administrators must test patches for each class of UNIX system, and then prepare a patch in the appropriate format for each system. For example, Solaris 8 and earlier use the patchadd command to install patches, while many Linux systems use some form of the rpm command. Once the patches are ready, the system administrators install the patch packages on server systems, and the client systems can fetch them from there. cfengine includes mechanisms for randomizing the time for requesting patches, preventing the network congestion that would occur if every system attempted to download patches at the same time.
The system administrators and IT department staff at SDSC don't spend all of their time patching. Instead, patching has become part of the weekly routine, and is executed smoothly enough and frequently enough to prevent problems. The difference between SDSC and many other organizations lies in two factors: One is the obvious goal of doing a good job at security and system administration. The second is the willingness of SDSC management to impose a policy on all their users?including researchers?which makes a successful patching strategy possible.
This discipline in installing patches has paid off. I visited SDSC in September 2002, as part of a project to find out how another research organization could improve its security. Some of SDSC's practices, such as running up-to-date anti-virus software on Windows systems, are the same as those of most businesses today. But other practices, such as the use of reference systems, evaluation and testing of patches, and routine installation of patches, make SDSC an example for the rest of the networking world.
But even if all organizations were to follow SDSC's lead, there would still be an enormous patching problem. That problem involves systems that belong to small businesses and home users, who rarely patch their systems. Failure to install patches is often due to a concern that installing a patch could break an otherwise working system. In other cases, it could be that the mere concept of downloading a Microsoft Service Pack that's just over 100Mbytes long over a not-so-reliable network link just doesn't compute.
Regardless of the reasoning, until the majority of systems are patched, none of our networks will be secure.
This NIST paper describes procedures for patch handling and includes
extensive lists of vendor resources:
http://csrc.nist.gov/publications/nistpubs/800-40/sp800-40.pdf
This BindView RAZOR article about patching focuses on deficiencies in digital signatures associated with vendor patches: http://razor.bindview.com/publish/papers/os-patch.html
To see NIST's searchable vulnerability database, go to: http://icat.nist.gov
For information on Purdue University and CERIAS' Cassandra tool for checking the
NIST vulnerability database, go to:
https://cassandra.cerias.purdue.edu/main/index.html