============================================================================= AL-95.01 AUSCERT Alert January 11, 1995 Ongoing Network Monitoring Attacks. ----------------------------------------------------------------------------- The following Alert message relates to a specific incident that occurred in early 1995. During this incident a number of sites were seriously attacked using a cookbook approach which gives the intruder root access by exploiting a number of well known vulnerabilities. The following information was released to help constituents to look for evidence of a similar attacks on their systems. Please be aware that the details are bound to change over time and that details of individual attacks will vary. The information provided below should be treated as general information only, to point you in the right direction. Locations of all files and programs mentioned in this document can be found in Appendix A. A list of SUN patches that relate to this incident can be found in Appendix B. 1. Description The intruders mainly target SUNOS 4.1.x systems, although other operating systems may be vulnerable to similar attacks. Therefore, the information given below mainly pertains to SunOS 4.1.x systems. Intrusions may be identified in other systems by using similar techniques, although the individual commands may differ. If your system has been compromised to root level, the intruders will most likely have modified system binaries and installed a network sniffer. You will need to look very carefully to spot these changes as the modifications are made to many binaries such as /bin/ls. These changes are designed to hide the intruders' activities. As the binaries for the commands used below may have been modified, you should execute the following commands using binaries taken directly from the distribution media rather than from the system under investigation. If the intruders have compromised a system on your network, they will use this system to compromise further systems at your site. This especially applies to systems referenced by NFS, NIS, and .rhosts or /etc/hosts.equiv files. 2. Detection Possible evidence of intrusion includes: 1. Files in /dev that are not device files but ASCII files. Particularly, /dev/ptyp, /dev/ptyq, /dev/ptyr. % /bin/ls -l /dev/pty? -rw-rw-rw- ... ^ note: '-' representing normal file instead of special file. 2. Trojaned ls(1), ps(1), ifconfig(8), netstat(8), login(1), plus others. These use the above files as configuration for what processes, files, and network connections to hide when executed. 3. An ethernet sniffer running. If ps(1) has been trojaned it won't show the sniffer process. Other (non-standard) system monitoring tools such as top(1) may show it as it runs. The sniffer process may be running with the same name as a legitimate system process. The command may look something like: -b -f The sniffer will be logging the data to disk. If your disk usage is steadily growing, this may provide some indication of the location of the sniffer logs. If you find a program executing that is similar to above, this may give an indication of the sniffer log file name. The sniffer must place your ethernet interface into promiscuous mode in order to operate. Some sites may already have this mode enabled, for example, to collect network statistics. It may be difficult to detect that your ethernet interface has been placed into promiscuous mode without authorisation, as ifconfig(8) may have been trojaned. The CERT Coordination Centre has provided a utility called cpm that can help determine if your ethernet interface is in promiscuous mode. cpm - check for network interfaces in promiscuous mode. The following is an example of the output of cpm and its exit status. Running cpm on a machine where both the le0 and le2 interfaces are in promiscuous mode, under csh(1): % cpm le0 le2 % echo $status 2 % Running cpm on a machine where no interfaces are in promiscuous mode, under csh(1): % cpm % echo $status 0 % If the intruders have installed a network sniffer, you'll need to find the logs. Forward these to AUSCERT. AUSCERT will need to contact all of the affected sites and advise them. Further information on network sniffers may be obtained from the CERT Advisory CA-94:01 Ongoing Network Sniffing Attacks. 4. Tracing ls(1) may give an indication of whether ls(1) has been trojaned. This will only work for SunOS. Under /bin/sh % cd /tmp % mkdir ls_tst % cd ls_tst % trace ls 2>&1 | grep open You should only see the following open messages (in order) open ("/usr/lib/ld.so", 0, 04000000021) = 3 open ("/dev/zero", 0 , 07) = 4 open ("/etc/ld.so.cache", 0, 05000000021) = 3 open ("/usr/lib/libc.so.1.8.1", 0, 045674) = 3 open ("/usr/lib/libdl.so.1.0", 0, 045714) = 3 open (".", 0, 0) = 3 If you see anything else (such as /dev/ptyr) then this may indicate that your ls(1) binary has been trojaned. 5. Tracing netstat(8) may give an indication of whether it has been trojaned. This will only work for SunOS. Under /bin/sh % trace netstat -a 2>&1 | grep open You should only see the following open messages (in order) open ("/usr/lib/ld.so", 0, 0101150) = 3 open ("/dev/zero", 0, 07) = 4 open ("/etc/ld.so.cache", 0, 0) = 3 open ("/usr/lib/libkvm.so.0.3", 0, 064460) = 3 open ("/usr/lib/libc.so.1.7.2", 0, 064500) = 3 open ("/usr/lib/libdl.so.1.0", 0, 064520) = 3 open ("/vmunix", 0, 0) = 3 open ("/dev/kmem", 0, 0) = -1 EACCES (Permission denied) If you see anything else (such as /dev/ptyq) then this may indicate that your netstat(8) binary has been trojaned. 6. Search the system for new or modified SETUID root files. # /bin/find / -perm -4000 -print Search the system for new or modified SETGID root files. # /bin/find / -perm -2000 -print If you are using NFS or AFS file systems, use ncheck(8) to search the local file systems. # /usr/etc/ncheck -s /dev/sd0a 3. Recovery If you believe your system has been compromised, the following steps should be taken: 1. Disconnect the affected machine from the network during the recovery. This will keep users and intruders from accessing the system. Keep a log book of everything you do. This will assist you in the future to understand the recovery procedure. 2. Notify AUSCERT (if you have not already done so). Contact information is at the end of this message. 3. Identify the extent of the compromise. This will involve identifying if a network sniffer is running and if any binaries have been modified. 4. Take a full backup of the system to save any evidence. 5. Advise the Australian Federal Police Computer Crimes Unit. The contact is: Det/Sgt Ken Day (03) 607 7777 6. If the intruders have installed a network sniffer, you'll need to find the logs. Forward these to AUSCERT. AUSCERT will need to contact all of the affected sites and advise them. If a sniffer has been operating, any network connections between two hosts that require the physical network segment that connects to the machine with the sniffer running, may have had the username and password details captured. This especially applies to telnet(1), rlogin(1), and ftp(1) connections. The intruders generally will zero a log file as they retrieve it. Therefore, you cannot assume you have a full list of compromised accounts. All vulnerable passwords should be changed immediately and the users instructed not to reuse their old password, ever. 7. For this type of compromise we recommend that you re-install the operating system from original media, rather than attempting to clean up. This should guarantee that all unauthorised modifications to system binaries are removed. You should install the latest available version of the operating system as many security problems associated with older versions are well known by the intruders. Sun has released some new versions of their operating systems recently. Contact your vendor for details. 8. Install all security patches pertinent to your operating system. Patches exist for the sendmail (see SERT Advisory SA-93.10), /bin/passwd, and /bin/mail vulnerabilities. A list of patches relevant to this particular incident is supplied in Appendix B. 9. If you run the elm mail handler, you should disable autoreply and arepdaem by typing as root: # /bin/chmod 700 /usr/local/bin/autoreply # /bin/chmod 700 /usr/local/bin/arepdaem 10. Include calls to a routine such as cpm (see above) and tripwire(8) (see below) in your regular system security checks. 11. Identify any back doors the intruder may have left to allow them further access to your system. These may include (but are not limited to): a) SETUID and SETGID files b) World or group writable files or directories c) Commands in the system startup and shutdown files d) Commands in .forward files e) Unauthorised .rhosts files f) Changes to system configuration files such as inetd.conf, program initialisation files for any user, mail aliases, hosts.equiv, hosts.lpd g) Files executed by root, but not owned by root h) Files and commands run by cron(8) or at(1) for all users (especially root). Also verify any files executed indirectly by cron(8) jobs i) Changes to group membership in /etc/group j) Any files (especially hidden files) with unusual names such as ".. " or "..^G" Note that tools such as COPS and TIGER check for signs of intrusion such as these. See Appendix A for details on where to get these tools. 12. Change the password on all accounts and instruct users not to reuse any old passwords. 13. Install the TCP Wrappers and configure them to allow access to your system only from authorised hosts. Note: If you are already running the TCP Wrappers, you should reinstall them. Verify that the configuration file names and contents have not been tampered with (for example, the hosts.deny file being renamed to "hosts.deny " - note the trailing space). 14. Install and configure tripwire(8) on your newly rebuilt system to take a snapshot of its present state. The configuration files, tripwire database, and tripwire binaries should be stored on hardware write-protected media (for example, a floppy disk). The system should be compared against the tripwire database using the safe binaries to verify any changes that have occurred since this installation. 15. Reconnect to the network and monitor your system for any further intruder activity. Don't trust your backups for reloading any file used by root. Doing so might re-introduce files altered by an intruder. If the cleanup is not performed correctly and thoroughly, the intruders will be able to compromise your systems again. AUSCERT provides a security checklist (SERT Advisory SA-93.06) to help you identify security weaknesses or modifications to your systems. --------------------------------------------------------------------------- Appendix A: Referenced files and programs SA-93.06 Unix Security Checklist Location: ftp://ftp.auscert.org.au/pub/auscert/advisory/ Filename: SA-93.06.UNIX.Security.Checklist SA-93.10 Sendmail Vulnerability Location: ftp://ftp.auscert.org.au/pub/auscert/advisory/ Filename: SA-93.10.Sendmail.Vulnerability CA-94:01 Ongoing Network Monitoring Attacks Location: ftp://ftp.auscert.org.au/pub/cert/cert_advisories/ Filename: CA-94:01.ongoing.network.monitoring.attacks.Z TCP Wrappers Simple Host Firewall System Location: ftp://ftp.auscert.org.au/pub/cert/tools/tcp_wrappers/tcp_wrapper.6.3/ Filename: tcp_wrappers.tar.Z Tripwire Simple File Integrity Checker Location: ftp://ftp.auscert.org.au/pub/coast/COAST/Tripwire/ Filename: tripwire-1.2.tar.Z cpm Check Promiscuous Mode Utility Program Location: ftp://ftp.auscert.org.au/pub/cert/tools/cpm/ Filename: cpm.1.0.tar.Z COPS and Tiger These packages identify common security and configuration problems. They also check for common signs of intrusion. Though there is some overlap between these two packages, they are different enough that it may be useful to run both. Both are available via anonymous ftp. COPS: ftp://ftp.auscert.org.au/pub/cert/tools/cops/1.04 tiger: ftp://ftp.auscert.org.au/pub/mirrors/net.tamu.edu/tiger* --------------------------------------------------------------------------- Appendix B: SUN Patches to install (that relate to this incident) Many of these patches and a full patch list are available via anonymous ftp from: ftp://ftp.auscert.org.au/pub/mirrors/sunsolve1.sun.com/ To obtain other patches not available via anonymous ftp, contact your local SUN Answer Centre for information. SOLARIS 1.1 recommended 100224-13 SunOS 4.1.1,4.1.2,4.1.3: /bin/mail jumbo patch security 100377-18 SunOS 4.1.1, 4.1.2, 4.1.3: sendmail jumbo patch 100383-06 SunOS 4.1.1,4.1.2,4.1.3: rdist security and hard links enhancement 102023-03 SunOS 4.1.3: Root access possible via forced passwd race condition SOLARIS 1.1.1 recommended 101436-08 SunOS 4.1.3_U1: patch for mail executable security 101665-03 SunOS 4.1.3_U1: sendmail jumbo patch 102060-01 SunOS 4.1.3_U1: Root access possible via forced passwd race condition SOLARIS 2.2 101842-01 SunOS 5.2: sendmail jumbo patch - security SOLARIS 2.3 101739-04 SunOS 5.3: sendmail jumbo patch - security 101574-04 SunOS 5.3: /usr/bin/mail jumbo patch SOLARIS 2.4 rev update 102066-02 SunOS 5.4: sendmail bug fixes SOLARIS 2.4.x86 rev update 102064-02 SunOS 5.4_x86: sendmail bug fixes --------------------------------------------------------------------------- If you have any doubts about the security of your system, or questions regarding this information please call AUSCERT. Internet Email: auscert@auscert.org.au Facsimile: (07) 365 4477 AUSCERT Hotline:(07) 365 4417 AUSCERT personnel answer during business hours (AEST GMT+10:00). (On call after hours for emergencies). Australian Computer Emergency Response Team c/- Prentice Centre The University of Queensland Qld. 4072. Australia