Network Defense--Incident Response Teams By Rik Farrow and Richard Power Last month's column examined hackers and some of their tools. Hackers have many sophisticated tools at their disposal, always for free, and many with user interfaces making these tools available even to the braindead. Now let's take a look at the other side of the wall--the defender's side. Quite often, the attackers are better prepared than the defenders. The attackers, after all, most likely have a goal and a plan. You, representing the defenders, might have a goal of not being attacked, but you probably do not have a plan to carry out in the event of an attack. To say the least, this is not good. Creating your plan will take the support of management, the creation of policy, and choosing a team to handle incidents. You will also need to educate your user community about their role when a possible incident is discovered. None of this is easy, but all of it is necessary. More and more sites today are preparing for the eventually of security incidents. It is not a question of if, but rather of when your site will be the target of an attacker, or an internal incident. Higher Authority Your first step should be to garner the support of management. In the best of all possible worlds, your management has approached you and asked for your assitance in creating an incident response team and support for the team. But it is much more likely that you are the sole person concerned with network security, and that you are already overworked and understaffed for the job you have been asked to do. You will see that having management support and creating a team will make your job easier when an incident occurs. The yearly CSI/FBI Computer Crime survey (http://www.gocsi.com/) is an excellent starting point. This report shows that reported incidents have increased over the three years of the survey, and that the dollar amount of each incident has also increased. Being able to quantify the potential cost of an incident will support any argument you might make in taking the time to set up policy and create a team. You can also begin to craft a risk assessment. Most IS shops have risk assessments in hand for their mainframes, but this excercise is seldom done for network servers, data warehouses, and client desktops. As mainframes are connected to your network, you should include them in your risk assessment. You can look at how the figures were created for the mainframe, and use this as a guide for creating assessments for your other key servers. For example, how much would it cost your organization if a key file server has to be re-installed from scratch on Monday morning? Some attacks destroy data, while others have left behind back doors and trojan horses, and re-installation is often the best solution. But what will the users of the server do while the system software, and the most recent backups are being re-installed? If twenty people use that server, and each makes $15/hour (not including benefits), then it costs $300/hour for lost time. Also include the time spent while re-installing the system and software in your calculations, as well as all the time for all the people involved. The risk assessment is not only a tool for convincing management, but it also helps you to become aware of exactly what systems are most important in your networks. In the event of a disaster, this preparedness will come in very handy, because you will have mapped out the critical systems, and know which systems must be restored first. New Policy In informal surveys taking during workshops and seminars, it appears that more than half of all sites already have a written security policy. Among CSI members' organizations, the number is closer to 90%. Policy is your most important tool in managing site security. Everything else you do is an implementation of the written policy. So you want policy also to cover incident handling. Working with your management and legal counsel, amend your policy (if necessary) to cover the duties of users, network and system administrators, and the response team. Users and administrators' single most important task is to know what to do in the case of a possible incident. Note that we said possible--it is not up to the user to determine if an incident has occurred. Some incidents will be blatant, such as theft or e-mail harrassment. Others could be a lot more subtle, such as truncated log files or modified system files. Your policy should provide a list of events which should launch a call to the team. Users and administrators must know more than who and when to call. Policy should dictate what to do in the event of an intrusion discovered while still in progress. In most cases, it is best to monitor the intruder, using a network sniffer or a tool such as IP Watcher (Engarde Systems). By watching the intruder, you may learn which systems have been infiltrated, what techniques have been used for breaking in, and information about the scope of the attack. Logging an intruder out leaves the intruder free to come back--unless you can be certain you closed all possible holes. Also, users and administrators must be taught to leave the "crime scene" alone. Imagine that you have discovered a murder in a hotel room. Would you first call the hotel and have them clean up, then call the police? Your users must leave systems untouched, and administrators must make only the minimal changes necessary to keep services available. System configuration is evidences, and changing it cleans up after the crime. The Team An Incident Response Team goes beyond the small group of people (perhaps the one, parttime person) who manages network security. You must have people with expertise in every operating system, major application, and network device on call to investigate or recover from an incident. The members of the team can change from time-to-time. Some FIRST teams rotate membership so that the tasks of participating never become too onerous. And you must have one person always available as the point of contact. Some organizations use their help desk as the point of contact for a potential incident. The help desk doesn't actually handle investigating the incident, but can log the request, provide a trouble ticket number, and contact the point person. You will want to have more than one person who can act as the primary contact. If you only have one person, what will you do if there's an incident and the contact person is sick, or at a security conference? The initial contact should be made by phone, in case e-mail has been compromised. The first step in investigating an incident is similar to triage-- quickly assimilating the scene and deciding what should be done immediately, and what can wait. Among the investigator's first acts will be the notification by phone of other people who should be involved--both network administration and managment. The investigator will also have to decide, if the incident is still in progress, to shutdown the system or to monitor traffic or keystrokes. After the triage phase, the investigator needs to take steps to prevent the incident from happening again. This may include disabling or reconfiguring software, alerting vendors of software bugs, and also sharing the information about the attack with other teams so that they can make appropriate preparations. If the incident requires a modification in policy, changes to policy will most likely have to be deferred until the incident is closed. The investigator must also act to preserve any evidence. With computer systems, this usually consists of logs and other files on hard drives. For desktop systems, the best way to preserve evidence is to clone the hard drive, and return the copy, keeping the original for evidence. Ideally, the compromised systems are shutdown as soon as possible, and their hard drives cloned or backed up. Organizations that are serious about security keep spare drives and cloning hardware on hand. The Law If you think that criminal prosecution is possible, you should contact the local or state district attorney's office. They will want to know what was done, as well as the amount of damage. Unless you can claim financial damage, it is unlikely that any prosecution will occur. But let the DA make that decision. Once law enforcement begins investigating, you must defer to their lead. You are still important to the investigation, but you are not the police. So the team becomes a resource for the investigation, but acts at their direction. As your side of the investigation proceeds, keep a log of everything that you do. Use the trouble ticket assigned by the help desk to organize your record. Record the time and date, location, detailed informatiob about anybody you talk to, including their names, phone numbers, email addresses, user names, even past employers. The more you write down, the better, as it may take over a year before a case appears in court, and good records will be your only hope of remembering clearly what happened. If the incident is an internal affair, you will be conducting the investigation. You or team members will need to interogate the participants. Ask direct, pointed, questions, and keep good notes. It is often a good idea to include an impartial witness--but warn all participants to keep the incident confidential. At the close of the incident, you may want to take further actions to prevent the recurrence. This is a chance for reviewing policy and procedures, how the incident was handled, and writing a final report. Remember that survivors are those that have a plan. Don't wait until an attacked finds you. Ready your defenses now. Resources: Icove, D., Seger, K, & VonStorch, W., Computer Crime, ISBN 1-56592-086-4, O'Reilly and Associates, 1995. Cavazos, E., and Morin, G., Cyberspace and the Law, ISBN 0-262-53123-2, MIT Press, 1994. Smith, D., Forming an Incident Response Team, ftp://ftp.auscert.org.au/information/papers.html RFC 2196 (also FYI 8), Site Security Handbook, http://rs.internic.net/nic-support/fyi/rfc2196.html FIPS documents from NIST: http://csrc.ncsl.nist.gov/ The FIRST (Forum of Incident Response and Security Teams) Web site: http://www.first.org/ The CERT Cordination Center: http://www.cert.org/ CIAC Security Web site: http://www.llnl.gov/ FEDCIRC Home page: http://fedcirc.llnl.gov/ [end resources] [sidebar steps] Steps toward efficient incident response Change policy to support incident handling Inform every user of your computer systems and network what to do if a possible incident has occurred: o use the phone to contact the designated person or agency, o avoid doing anything which might damage evidence, and o not to contact the press or media Make everyone aware of what constitutes a potential incident: o violations of the security policy o evidence of unauthorized use, such as missing or truncated log files, log entries recording logins at strange hours or by supposedly dormant accounts, changes to system configuration, changes in authentication sequences, other unusual events Plan in advance a strategy for handling an incident-in-progress; o circumstances when the attack should be terminated at once, or monitored o collect tools and practice monitoring network connections--without violating your organization's policy regarding privacy Create a team for incident handling, including people competant in every network or operating system type in your organization; rotate team members to prevent burnout Prepare for evidence collection by learning how to clone hard drives, and (if possible) by keeping spare drives on hand; keep hard copy notes of everything that you do or learn If you believe a crime has occurred, contact your state or federal district attorney; be ready to present a statement of damages, including the cost of lost work, time spent investigating and restoring systems, value of proprietary information which may have been stolen, etc. At the close of an incident, review your handling of the incident [end sidebar] Richard Power is editorial director of the Computer Security Institute (San Francisco). You can reach him at rpower@mfi.com. Rik Farrow is an independent security consultant. You can reach him at rik@spirit.com.