by Rik Farrow <rik@spirit.com>
When someone with half a clue decides to attack your system, they will first try to identify the operating system. Not every attack proceeds this way -- script kiddies will probe huge address spaces looking for any system with a particular port open, indicating that just maybe that system will be vulnerable. But for the professional penetration tester or hacker, operating system (OS) identification is an essential step in probing.
My September 2000 column considered using ICMP packets for probing, as well as detecting operating system types. Since that article was written, Ofir Arkin has updated his ICMP paper, including new tricks for OS identification. But using ICMP will not always work, as most firewalls will block some (or all) incoming ICMP packets. That leaves the classical methods, as well as the newer techniques known as stack fingerprinting.
The king of the stack fingerprinting programs today is nmap. I had often wondered exactly how Fyodor, the author of nmap, went about designing his program. In his paper, Fyodor mentions TCP options, and how useful these options are for OS identification via stack fingerprinting, and this column goes deeper into TCP options, and how they are used by TCP/IP, and nmap.
The easiest way to identify an OS is to contact it using some open port. While this takes a little finessing (not much), most telnet, email, FTP, and Web servers will identify not only themselves, but also the operating system on which they are running. This is almost too easy, although it can sometimes be unrealiable--cagey administrators may change the login banners to mislead an attacker.
Other strategies for identifying operating systems include a port sweep and grabbing email headers. Port sweeps identify open ports, and on a system that has not been hardened, just this list of open ports is enough to identify most operating systems. Email headers can identify mail user agents, the user's operating system, the mail server, and sometimes even the firewall that the mail passed through. The amount of information that can be gleaned from email is quite amazing, and perhaps the topic for another column.
The easiest way of OS identification has become running nmap. Nmap, or network map, started off as a very functional network and port scanner, but has grown over the years to become the most feature rich free stack scanning tool in existence today. Like many tools available on the Internet, this one is just as useful whether you are wearing a white hat or a black one. Two years ago, Fyodor, author of nmap, added OS fingerprinting techniques. Looking in the fingerprints file of a recent version of nmap (2.53), I counted fingerprints for 465 different stacks, including operating systems, routers, printers, and other networked devices.
The fingerprint file (nmap-os-fingerprints) provides some clues in the comments at the head of the file: there are nine different tests, each with subparts, for identifying an OS, although the order presented is a bit misleading. For a deeper look, you can try reading the source for the OS fingerprinting module yourself, osscan.c, as well as running nmap -O and monitoring your network so that you can see what it is doing. Besides getting a better understanding of nmap, going through this exercise provides you with ideas for recognizing it on your networks, as well as blocking its actions.
For example, suppose you were to build nmap, and then run it against a known target while you capture the packets it outputs, as in nmap -p20-23 -O target. The -p20-23 limits the number of ports scanned (be sure you include an open port in this range), and the -O enables the OS identification feature. Nmap (after a DNS lookup if required) goes through four phases: ping, port scan, characteristics collection, and sequence number analysis.
Nmap begins with two tests to see if your target is up and responding. It first sends an ICMP Echo Request (ping), and then probes port 80 (the Web server port), just to see if the target is alive. These initial probes can be disabled or modified with nmap options, so they are not very reliable indicators of an nmap scan in action.
Next, nmap will port scan, in this case, looking for open and closed ports
in the 20-23 range. The target system in this example had port 23
(telnet) open,
which is important for the next phase of scanning. Once nmap has
identified one open and one closed port, and it can begin the actual
OS identification portion. Nmap will send seven carefully crafted TCP
packets to the target, four to the open port, and three to a closed
port, then sniff the responses as they come back. The responses
are parsed and stored in a format that matches the entries in the
fingerprint file.
Each of the packets has a different combination of TCP flags sent, with the goal being to tickle out an OS identifying response from each. For example, the first packet to an open port includes only the SYN flag, typically used when opening a TCP connection. The second packet includes no flags (a null scan), and the third probe to the open port includes the flags SFPU (SYN, FIN, PUSH, and URGENT), an unusual, but not illegal set of TCP flags. In RFC 1025, this set of flags with one byte of data was called a Kamikaze packet, and was used in testing of TCP/IP stacks. It is this peculiar set of flags that Snort, an open source ID system, uses to detect nmap OS detection scans. Another trick for picking up these probes is that the same TCP sequence number is used for all seven of these probes.
Besides just sending packets with different TCP flag settings, these packets also include several TCP options, something that I had noticed before in network monitor output, but not really understood. Digging into the RFCs, I found several references to TCP options, but the most important for understanding what nmap is doing was RFC 1323. Rich Stevens' TCP/IP Illustrated, Volume I, made interpreting the options easy (pages 253-254).
First, you can tell that options are present in a TCP header by examining the length field. This field is normally five, but if it is greater, than some options are present at the end of the header. Options begin with a 'kind' field that identifies the type of options. Only the no-operation (NOP, an option used for padding) and the end of list options consist of a single byte. All other kinds of options consist of the kind, a length in bytes that includes the kind byte, and the actual data. The NOP is used to align the data in options into 32 bit words, or to fill out a 32 bit word. The end of list always ends the options.
Fyodor uses three options, plus a nop for padding and the final end of list. The options are window scale, maximum segment size, and timestamp. Of these, window scale and timestamp were added (in RFC 1323) to support high speed networks with relatively long latency, or to quote Van Jacobsen, Long, Fat Networks (LFNs). The MSS or maximum segment size is normally used by the sender to avoid fragmentation. In nmap, the order of these options is significant, as the fingerprints record the order that the target of the scan returns the options. These options should be sent first in an initial (SYN) packet, a rule that nmap ignores in several tests.
Some TCP/IP stacks do not understand any options. Others, will respond with some or all of the options, or even include additional ones. Nmap records these reponses, along with other characteristics, such as where padding (nops) is used, is the Don't Fragment flag set, the size of the Window value in the TCP header, and if the acknowledgement number is set to zero, incremented, or left unchanged from the sequence number sent.
The Window is a flow control mechanism used in TCP connections. By sending a Window value, the sender tells the receiver how many bytes of data can be sent without receiving an acknowledgement. This keeps one system from sending data to an overloaded and slowly responding machine, which can also stop the flow of data by setting the Window value to zero. Although it may seem logical that all systems use the same Window value, the initial Window value varies enormously, from 0 for some devices, to 32767 for Windows NT/SP3 and just one Linux kernel (2.1.76). Although there many be several systems with the same initial Window value, nmap uses other characteristics to select the OS type, breaking the tie.
Nmap also collects characteristic from a response sent to a closed UDP port (which will be an ICMP Destination Unreachable, type Port Unreachable), as the target is supposed to include part of the packet that caused the ICMP message. The accuracy of the quoting of the packet in the ICMP response varies depending on the target stack, and this too can be used to identify an OS.
The final stage uses SYN packets to probe for initial sequence nubmers. Nmap sends a couple of resets first to the open port, then sends six packets with just SYN set (the normal method for opening a TCP connections), followed each time with a reset (TCP header with Reset and ACK flags set, aborting the connection; these resets are actually sent by the stack as a response to receiving the acknowledgments to the probe packets). The sequence numbers in packets sent get incremented by one each time, abnormal behaviour, but characteristic of sequence number collectors, such as rbone and the unpublished tool used to takedown Tsutomo Shimomura's site back on Christmas Day, 1994.
Nmap collects the initial sequence numbers received from the target, and looks for a pattern in the way they are incremented. Really old UNIX systems still use a constant increment, while newer and more secure systems use a random increment. Newer Microsoft stacks use a time-dependent increment, which might make them vulnerable if they ran a UNIX service such as rlogind (which is not the case).
Nmap's accuracy relies on collecting a lot of information from the target's stack. Anything that tampers with this information will affect the prediction of the target's OS version. What might tamper with this? Firewalls do to a greater or lesser extent. Most stacks permit tuning some of the tested parameters, but only a small portion of them.
Packet filtering firewalls are ineffective. Firewalls that perform Network Address Translation will be a little bit better if they rewrite sequence numbers (not all of them do). Application gateways running on a firewall prevent nmap from identifying the target, as nmap will be probing the firewall's stack instead. Of course, the best defense is to prevent any scanning from penetrating your network in the first place, and detecting scanning if it takes place within the confines of your network.
The home site for nmap: http://www.insecure.org/nmap/; Fyodor's paper describing the techniques he uses for operating system detection: http://www.insecure.org/nmap/nmap-fingerprinting-article.html (also included with source for nmap).
RFC describing TCP Extensions for High Performance with two of the three TCP options used in OS detection: http://www.faqs.org/rfcs/rfc1323.html
Snort, an ID scanner for low volume networks: http://www.snort.org/
New version of Ofir Arkin's paper (see September 2000 Network Defense
column) on ICMP used in scanning:
http://www.sys-security.com/archive/papers/ICMP_Scanning_v2.0.pdf
Windows NT version of nmap at eEye:
http://www.eeye.com/html/Databases/Software/nmapnt.html
Richard Stevens, TCP/IP Illustrated, Volume 1, Addison-Wesley.