by Rik Farrow <rik@spirit.com>
Around midnight of November 7, 2000, a Linux server was broken into. In and of itself, this event was nothing special -- systems get hacked all the time. What made this attack different were the forensic dissections that followed.
The Linux system in question had been setup by the Honeynet Project, (http://project.honeynet.org), a coalition of security professionals interested in learning and in sharing what they learn about scanning and attacks. To this end, an ID (Intrusion Detection) system was set up to collect network traces of scans and attacks, and monthly challenges, along the lines of "identify the scanning tool" or "name that hack". When a Linux system was attacked and thoroughly subverted only two and a half days after it was installed, the system itself became The Forensic Challenge.
The official results of the challenge can be viewed at the Honeynet Project Web site (see Resources). In this article, I go through my own process of analysis of the attack, and what the attacked installed or otherwise modified the victim system, with the added insights of Dave Dittrich, the Senior Security Engineer at the University of Washington. What the attacker did after the initial exploit is even more interesting than the attack itself.
When a system gets hacked, the system administrator or security consultant usually attempts to determine how the attack occurred, and patch the system. In many cases, no effort at all is made to determine why the attack succeeded--the system is simply re-installed. If the method of attack has not been determined, the newly installed system is likely to be hacked again, and soon.
The Linux system belonging to the Honeynet Project was destined to suffer a very different fate. Shortly after the attack, the system (named apollo) was taking down quickly, preserving the hard disk in exactly the state it had been in shortly after the attack. Each disk partition was copied byte-for-byte (using the UNIX dd command) to files, and the files posted on the Internet. Altogether, the compressed image copies of these disk partitions compromised 1.5 gigabytes of data (three gigabytes when uncompressed).
Not very long ago, the idea of downloading a complete set of partition images and manipulating them locally would have been impossible. But disks are now cheap, and building a system that is configured like the victim, a RedHat 6.2 default server install, is easy to do. Extracting the partition files took just three minutes; uncompressing the gzipped images took twenty minutes. Both Linux and BSD-style UNIX systems permit mounting partition images as if they were real paritions (using the loopback interface), so the images from the victim system could be locally mounted as read-only file systems.
The Challenge home page provided the IDS traces of the probes and the attack (see Resources). Someone scans apollo, the victim, looking at the portmapper, the RPC (Remote Procedure Call) port maping service at 23:11:06. Forty-five seconds later, immediately after another portmapper request, shellcode is sent to port 871/udp on apollo. Port 871/udp is unassigned, but logically belongs to some RPC service found in a standard Linux server install.
Snort, a free IDS tool, was used for monitoring the Honeynet. Snort also captured the shellcode used, binary code sent when exploiting a buffer overflow or a format string bug. The shellcode shows that the Internet daemon configuration file was modified, adding a new service to port 4545/tcp that grants an immediate, root-owned shell to anyone who connects to that port.
Getting to this point from just the snort traces was easy. Figuring
out exactly which service on apollo had been exploited was much harder.
For my analysis, I started looking at the configuration files, trying
to determine which service would be listening at port 871/udp.
I couldn't find any service that matched, but I did
discover that the script that starts the system logging daemon (syslogd)
had been deleted. Digging deeper into the system startup scripts hinted
that only two RPC services would be started on apollo, statd and ruserd,
and of the two, statd is well known for having been exploited. Using a
byte pattern in the shellcode, I used Web search engines to scan for
any exploit sources that matched this attack, but found no exact matches.
.sp
Dittrich took a different approach. He had recovered a fragment of a
log file in a deleted file that contained an log message from statd
when it was exploited, and matched this error message to one found
in CERT's vulnerability database.
I also discovered a directory named ' ' (/usr/man/.Ci/ ). Directories with funnier names than usual are excellent indications of hacking activity, but all I found in this directory was a single script which did little. The parent directory, also an unusually named one (.Ci), was a different matter.
I had not gotten very far in my analysis, as slogging my way through the nearly 30 thousand files and directories in a default Linux server install is a tedious process. It was high time to install The Coroner's Toolkit (TCT), a set of C programs and Perl scripts written by the team of Wieste Venema and Dan Farmer (once famous for writing SATAN).
At this point in time, TCT works best on Linux, BSD, and Solaris systems. Some portions of the tool, like the extremely useful mactime script, also work on FAT and NTFS filesystems, while others only work on the Linux ext2fs or UNIX file systems (ufs). Installing TCT took only minutes, but figuring out what to do next took longer.
Venema and Farmer have written a series of articles (which you can find through their Web sites) about forensics, and reading these is the recommended approach for using TCT. I also read their on-line docs, then fired up grave-robber, a script that collects information about a system, and saves it, along with MD5 checksums (useful if you later plan on presenting your research as evidence) of everything about the victim system.
While grave-robber's output proved interesting, what was more useful was the mactime script. The mactime Perl script processes some of the output of grave-robber, the body file, and creates a listing showing when files were read, modified, or had their metadata changed. All UNIX file systems and NTFS include this information about every file and directory. Mactime sorts this information into chronological order, which is magic indeed when you are attempting to understand what happened on a system in the not too distant past.
TCT also includes unrm and lazarus. The unrm program scans free space, collecting blocks that are not allocated to any files or directories. Whenever a file is deleted, it becomes part of free space, and if the system is not very active, it is possible, although difficult, to recover deleted files from free space. Running unrm was fast. I also tried lazarus, which attempts to organize the gigabytes of recovered free space into blocks containing text, email, tar archives, C source, or executables by examining the content of each block. Lazarus took over 9 hours to exhume the /usr partition. I found unrm and mactime much more useful, although the blocks of text created by lazarus can be searched as well.
I also learned a great trick from reading Dittrich's analysis. You can combine unrm with ils (another TCT tool) and recover data about deleted files from each partition, and fold this information into the 'body' file created by grave-robber. Then, when you run mactime against the 'body', you see not only existing files and directories, but also ones that have been deleted.
Because apollo had been halted the evening after it had been hacked, most traces were still around, and running mactime revealed quite a lot. It was like visiting a crime scene where there was fresh mud all around, and lots of footprints and fingerprints everywhere, just waiting for Sherlock Holmes to arrive.
By looking at the mactime output, you can tell that
the attacker logs into apollo,
using the backdoor installed in inetd.conf, at 6:25 Central Time,
executes uptime (and perhaps some other commands), deletes
hosts.deny (used in access control), and then uses FTP to
start downloading tools. Once the tools are downloaded,
the very first thing the attacker does is to unpack and
install eggdrop, an IRC bot used to control IRC channels.
This action indicates that the reason for this attack was in
support of IRC wars.
Just as interesting, the next thing the attacker does is to unpack scripts and exploits into the /usr/man/.Ci directory. These scripts are used to strip logfiles, add new user accounts, and install trojan horses and backdoors. For example, the user accounts own and adm1 are added to the system, and used during this phase, and the initial backdoor, the line added a shell to the Internet daemon configuration file (inetd.conf) is removed. To prevent logging, the wtmp log file is deleted, the system and kernel logging daemons halted, and shell history files for the new user accounts (.bash_history) linked to /dev/null (the UNIX bit bucket).
The attacker disconnects, then logs back in using one of the new accounts, and then changes over to an existing user account (drosen). Under the drosen account, the attacker installs the IRC bot, as well as the other scanning and exploit tools.
The next order of business is to patch the system. Obviously, the attacker doesn't want anyone else breaking into this insecure system. A script installs various gzipped archives and RedHat patch modules (RPMs) to update rpc.statd (the hole the attack used), as well as wu-ftpd, the automounter, named (DNS software), Kerberized telnet, LPR (lineprinter software), and some other RPC services.
The attacker also installs a trojanned and backdoor version of SSH, the Secure Shell, version 1. The normal version of SSH provides strongly authenticated and encrypted communications, which is great for remote administration. The hacker's version does this, and more. For example, anyone using the trojanned version with a password (instead of using a digital signature) will have their password logged to the file /var/tmp/nap. The trojanned SSH also has a backdoor, granting root access to anyone who knows the magic password. The trojan also logged the 'magic' password, tw1l1ghtz0ne, in the same logfile.
Once SSH is installed and running, the attacker logs out, and logs back in (twice) using SSH. The new accounts own and adm1 are removed from /etc/passwd and /etc/shadow (user accounts files), and more trojans are installed. The attacker installs trojan versions of ifconfig, ls, netstat, ps, top, and tcpd to hide files, network connections, and current activity on apollo. Scripts and programs are run to add filenames, network addresses and ports, and program names to the configuration files used in hiding this information: ls uses /usr/man/r, ps and top use /dev/ptyp, netstat uses /usr/libexec/awk/addy.awk, tcpd uses /usr/man/.a, and ifconfig simply never reports an interface in promiscuous mode.
The trojanned ifconfig is for hiding snif, a version of LinSniffer, used to sniff passwords off the local network. Although snif was started (the log file appears), no passwords were captured. The trojanned tcpd, which normally provides additional logging for services started through the Internet daemon, also includes backdoors configured through /usr/man/.a. All of the trojans were installed using fix, a program that appeared in one of the earliest rootkits that I have seen (one written for SunOS in 1993).
The attacker took approximately one hour and twenty minutes, some of that time spent with two connections, thoroughly setting up the victim for future use. The various scanning software was designed to sweep through Class B-sized address spaces (65 thousand subnets at a time) looking for the vulnerabilities the attacker had patched.
The people analysing the traces of the attack took thirty hours or more each. What I learned from this exercise was the usefullness of TCT, as well as the importance of IRC, trojans, and scripts to the attacker, as well as his paranoia about being hacked.
The HoneyNet Project's home page, with links for the Forensic Challenge: http://project.honeynet.org; you can find the original challenge, as well as snort IDS output here.
Dave Dittrich's home page, with loads of security relevant links: http://www.washington.edu/People/dad/
The entry matching the attack pattern that Dittrich found in CERT's vulnerability database: http://www.kb.cert.org/vuls/id/34043
The Coroner's Toolkit can found at Wietse Venema's site, under Forensics: http://www.porcupine.org/forensics/, or at Dan Farmer's site: http://www.fish.com/forensics/