Penetration testing works best as an audit of your security policy and procedures
by Rik Farrow
During a conversation with a client, I heard that they planned to hire a company that specializes in networking to run a vulnerability scanner to test their firewall. The client called this process "penetration testing", but I was appalled. My own concept of penetration testing was very different from what they had in mind.
I have never performed "formal" penetration tests, as I believe that is best done by a group of people with varying specialties. I have performed less formal tests for clients and friends using port scans and tests of the services open at particular ports. But after checking with other security consultants, I discovered that what I had been doing could be called penetration testing. And the notion of paying someone to run a vulnerability scanner against a site turns out to be not as farfetched as I had first thought, although other experts in this field recommend doing a lot more than just running some tool, handing over the report it produces, and delivering an invoice.
Penetration testing does provide real value for any organization interested in network security. But finding the right company to perform a penetration test is not easy. And getting an effective penetration test also relies on you, the target, working with the penetration testers.
My own personal image of penetration testers was created by the movie Sneakers (http://www.allwatchers.com/topics/info_13234.asp), where a group of security experts used a variety of techniques to penetrate first a bank, and later a security company. The techniques involve social engineering, use of a a video camera with a long lens to watch passwords being typed, wiretapping, theft, as well as technical tricks. Of course, the real world of penetration testing is very different.
Penetration testing provides a mechanism for proving that your security works the way that you want/expect it to work. Let's assume that your organization is already doing the right things: regularly updating your policy and procedures, keeping systems patched, especially any exposed or critical systems, and using security tools such as vulnerability scanners to see that your network really is fully patched. If you do all these things, why would you want someone else to perform an audit or penetration test?
Penetration testing provides an independent examination of your security, a second set of eyes. And not only eyes, but individuals whose entire professional lives revolve around looking for flaws in the security of networked systems.
Penetration testing may be part of an external audit. In particular, penetration testing refers to probing systems to identify the operating system and any network services, then checking for vulnerabilities in the network services found. You can do these things pretty well with vulnerability scanners, so why hire someone else to do them for you? Not only do you want third parties checking your work, you also want them to use different tools, and people who are familiar with using those tools.
In a review that appeared in Network Computing (see Resources), the authors discovered that none of the vulnerability scanners tested found all of the known vulnerabilites in a set of target systems. Of the seventeen vulnerabilities intentionally installed on the target systems, no scanner found more than fifteen, and none of the scanners found one of the vulnerabilities, even though it was one that was often exploited. Though this review is several years old (2001), I do not believe that the art of vulnerability scanning has improved that much--certainly not to the point of perfection.
So part of the art of penetration testing becomes interpretting the results of tools used during the probing process. Anyone who owns a vulnerability scanner can run the tool against your firewall, or portions of his or her network. But there are few people competant enough to understand fully the results of a vulnerability scanner, and actually capable of performing additional tests to prove that the vulnerability scanner's report is actually accurate.
Some penetration testers use two scanners to perform the vulnerability assessment. At first, the use of a vulnerability scanner seemed, to me, like a form of cheating. But such tools can automate at least part of the process, and allow skilled individuals to focus upon anything that appears to be a problem. Deeper probing should involve connecting to any suspect service, and, in some cases, actually attempting an exploit.
Another issue with commercial vulnerability scanning tools is that many products hide the results when a particular test fails to acchieve a definite answer. One well-known product, for example, would not report that a Cisco router was vulnerable to certain DoS attacks if the scanner could not log into the router, or use SNMP to obtain the software version. If you don't know that a scanner hides the information (that it failed to test for the vulnerability), you could easily be fooled into believing that your network is safe, when, instead, your vulnerability scanner product has avoided telling you that the true state of part of your network is unknown.
Besides finding a competant organization to perform a penetration test (see audit article in the Resources), your organization has its own tasks to perform. First among these is to define the scope of any testing.
Sometimes management will suggest a "black box" approach to penetration testing. In black box testing, the penetration testers are told nothing about the target, under the assumption that real attackers will work under similar conditions. Not only is this a good way to waste your organization's money, it is also not true. Attackers may garner information about your organization through social engineering, theft, bribery, and breaking and entering. Real attackers will not be limited to attacking your own organization, but might also break into other organizations or your ISP. Penetration testers will be (should be!) law abiding, and operate under a very different set of restraints.
To get the most out of a penetration test, you want to provide as much information as possible to the testing organization. Keep in mind that any testing organization should expect to sign a non-disclosure agreement, so that you can feel comfortable with sharing your policy, procedures, and information about your network and critical systems.
You want to decide which systems need to be tested. You don't want to exclude any system that could conceivably be attacked, although you might want to contract out your penetration tests in phases, focused on different portions of your network. You also must set down guidelines, for example, that the penetration experts can probe and test for vulnerabilities, but not actually exploit those vulnerabilities. Exploiting found vulnerabilities creates a lot of flash, but can endanger the very systems that you want to protect.
You also want to provide access for the test. If you want systems within your DMZ tested, the best place to test them is from within that same network. Forcing the penetration testers to work outside of your firewall might seem more realistic, but an internal test is much more likely to find flaws in server security that your firewall currently hides. These same flaws might later be exposed by changes in your firewall, or through exploits that make it possible to use one DMZ server to attack other servers. Just recall attacks like Slammer, that used a Web server to launch further attacks. While the best penetration testers might still succeed while working through your firewall, stacking the deck in favor of success will provide you with a much better payoff.
I was once hired to check out a firewall and the two systems protected by that firewall. As soon as I scanned the DMZ, I found three systems, instead of the two I had been told about. The third system had just been replaced by one of the two systems I had been asked to check. But it was that third system that contained the most vulnerabilities. And, in this case, the firewall was still configured to provide complete access to internal networks for any network traffic, or attacks, coming from the third system. Scanning within the DMZ made finding that system simple.
In the case of Web or application servers that can be accessed externally, you should also consider sharing the source code to these application to the penetration testers, if the scope of the work includes testing these scripts and/or programs. Testing ASP or CGI scripts without the source code is much harder to do, and deciding in advance that no attacker will ever see the source is not a good idea. Flaws in Web server software have exposed scripts and applications to remote attackers many times. Having access to the source of an application to be tested makes the process more efficient and effective. After all, you are paying the penetration testers to look for flaws, not to waste their time.
The point of penetration testing is to prove that your network defense works the way you thing it does. Quite often, system and network administrators consider auditors or penetration testers as the enemy, when in fact, they are allies. A good penetration test might prove your defenses really do work. Or, that you have issues that need to be addressed that you can fix before you get attacked successfully. It is much better to pay someone to discover your holes, than to have someone you don't know do it to you.
Penetration testing can also be used to provide concrete evidence to some third party, such as a financial backer or your own management. Sometimes, you will already know that your network defenses have problems, but you can't get management to allocate the resources to fix them. Just as a prophet is never recognized in his (or her) home town, having external consultants say the same things you have often works wonders with management.
You should include in any penetration testing contract or statement of work exactly what you expect in the report. Some consultants add a cover sheet to the report generated by the tool they used. If you paid for a minimal test, then a computer generated report can be expected. But the real value of any penetration test comes with the analyses that accompanies any report. The penetration testers must not only point out what they found, but also the significance of their findings. Where appropriate, the penetration testers can also suggest methods for remediation, for example, updating a server, disabling network services, changing firewall rules, etc.
Vulnerability scanners by themselves only scratch the surface. Penetration testing requires true competance not only in the use of such tools and the interpretation of their reports, but in knowing how to take the next steps to verify anything reported by a vulnerability scanner. The competant penetration tester uses one or more vulnerability scanners as no more than a part of their arsenal of tools.
Note that most attacks seen today perform only a primitive form of vulnerability scanning--the attack is attempted, and if it succeeds the target was vulnerable. An attacker who attempts to vulnerability scan your site will generate hundreds of firewall log messages, and any IDS will merrily begin announcing alerts about attacks in progress. Try it yourself, if you haven't already, by running a vulnerability scanner on a network watched by IDS. Most likely, you have already annoyed by someone running a vulnerability scanner without announcing it first. Note that you don't want to run the scanner if your policy forbids doing so. I don't want to suggest that you do something that might get you fired.
Penetration testing is another weapon in your network defense arsenal. Consider it as part of any security audit--but make certain that your auditors are up to the task.
The movie Sneakers: http://www.allwatchers.com/topics/info_13234.asp
Old (2001) Network Computing review of vulnerability scanners: http://www.networkcomputing.com/1201/1201f1b1.html
More recent review of vulnerability scanners: http://www.infosecuritymag.com/2003/mar/cover.shtml
Products and tools for vulnerability assessment of databases: http://www.networkintrusion.co.uk/database.htm
Open Source security methodology: http://www.isecom.org/projects/osstmm.htm
Carole Fennelly's audit article:
http://www.infosecuritymag.com/2003/mar/watchingwatchers.shtml