"Linux Gazette...making Linux just a little more fun!"

You've subscribed to Bugtraq and The Happy Hacker list, bought yourself a copy of The Happy Hacker, and read The Cuckoo's Egg a few times. It's been a very merry Christmas, with the arrival of a cable modem and a load of cash for you, so you run out and go shopping to start your own hacker lab. A week later, you notice that one of your machines is being an especially slow slug and you've got no disk space. Guess what - you got cracked, and now it's time to clean up the mess. The only way to be sure you get it right is to restore from a clean backup - usually install media and canonical source - but let's see what the "h4x0r" left for us to study.

In late October of this year, we experienced a rash of attacks on some workstations here at the University of Alberta's Department of Mathematical Sciences. Many of our faculty machines run RedHat 5.1 (there's a good platform to learn how to try to secure...) since it's cheap and easy to install. Workstations are often dual-boot with Windows 95, but we'll be phasing that out as we get Citrix WinFrame installed. This paper is an analysis of the compromise of one professor's machine.

One fine day I was informed that we'd just had another break-in, and it was time for me to show my bosses some magic. But like a skilled cardshark who's forced to use an unmarked deck, my advantage of being at the console had been tainted. Our cracker had used a decent rootkit and almost covered her tracks.

In general, a rootkit is a collection of utilities a cracker will install in order to keep her root access. Things like versions of ps, ls, passwd, sh, and other fairly essential utilities will be replaced with versions containing back doors. In this way, the cracker can control how much evidence she leaves behind. Ls gets replaced so that the cracker's files don't show up, and ps is done so that her processes are not displayed either. Commonly a cracker will leave a sniffer and a backdoor hidden somwhere on your machine. Packet sniffers - programs that record network traffic which can be configured to filter for login names and passwords - are not part of a rootkit per se, but they are nearly as loved by hackers as a buggered copy of ls. What wouldn't want to try intercept other legitimate user passwords?

In nearly all cases, you can trust the copy of ls on the cracked box to lie like a rug. Don't bet on finding any suspicious files with it, and don't trust the filesizes or dates it reports; there's a reason why a rootkit binary is generally bigger than the real one, but we'll get there in a moment. In order to find anything interesting, you'll have to use find. Find is a clever version of 'ls -RalF | grep | grep | ... | grep '. It has a powerful matching syntax to allow precise specification of where to look and what to look for. I wasn't being picky - anything whose name began with a dot was worth looking at. The command: find / -name ".*" -ls

Sandwiched in the middle of a ton of useless temporary files and the usual '.thingrc' files (settings like MS-DOS's .ini) we found '/etc/rc.d/init.d/...'. Yes, with 3 dots. One dot by itself isn't suspicious, nor are two. Play around with DOS for about two seconds and you'll see why: '.' means "this directory" and '..' means "one directory up." They exist in every directory and are necessary for the proper operation of the file system. But '...' ? That has no special reason to exist.

Well, it was getting late, and I was fried after a day of class and my contacts were drying up, so I listed /etc/rc.d/init.d/ to check for this object. Nada. Just the usual SysV / RH5.1 init files. To see who was lying, changed my directory into /tmp/foo, the echoed the current date into a file called '...' and tried ls on it. '...' was not found. I'd found the first rootkit binary: a copy of ls written to not show the name '...' . I will admit that find is another target to be compromised; in this case it was still clean and gave me some useful information.

Now that we knew that '...' was not part of a canonical distribution, I moved into to it and had a look. There were only two files; linsniffer and tcp.log. I viewed tcp.log with more and made a list of the staff who would get some unhappy news. Ps didn't show the sniffer running, but ps should not be trusted in this case, so I had to check another way.

We were running in tcsh, an enhanced C-syntax shell which supports asychronous (background) job execution. I typed './linsniffer &' which told tcsh to run the program called linsniffer in this directory, and background it. Tcsh said that was job #1, with process ID 2640. Time for another ps - and no linsniffer. Well, that wasn't too shocking. Either ps was hacked or linsniffer changed its name to something else. The kicker: 'ps 2640' reported that there were no processes available. Good enough. Ps got cracked. This was the second rootkit binary. Kill the currently running sniffer.

Now we check the obvious: /etc/passwd. There were no strange entries and all the logins worked. That is, the passwords were unchanged. In fact the only wierd thing was that the file had been modified earlier in the day. An invocation of last showed us that 'bomb' had logged in for a short time around 235am. That time would prove to be significant. Ain't nobody here but us chickens, and none of us is called bomb...

I went and got my crack-detection disk - a locked floppy with binaries I trust - and mounted the RedHat CD. I used my clean ls and found that the real ls was about 28K, while the rootkit one was over 130K! Would anyone like to explain to me what all those extra bytes are supposed to be? The 'file' program has our answer: ELF 32-bit LSB executable, Intel 80386, version 1, dynamically linked, not stripped. Aha! So when she compiled it, our scriptkiddie forgot to strip the file. That means that gcc left all its debugging info in the file. Indeed, stripping the program brings it down to 36K, which is about reasonable for the extra functionality (hiding certain files) that was added.

Remember how I mentioned that the increased filesize is important? This is where we find out why. First, new "features" have been added. Second, the binaries have verbose symbol tables, to aid debugging without having to include full debug code. And third, many scriptkiddies like to compile things with debugging enabled, thinking that it'll give them more debug-mode backdoors. Certainly our 'kiddie was naive enough to think so. Her copy of ls had a full symbol table, and source and was compiled from /home/users/c/chlorine/fileutils- 3.13/ls.c - which is useful info. We can fetch canonical distributions and compare those against what's installed to get another clue into what she may have damaged.

I naively headed for the log files, which were, of course, nearly as pure as the driven snow. In fact the only evidence of a crack they held was a four day gap. Still, I did find out something useful: this box seemed to have TCP wrappers installed. OK, those must have failed somehow since she got in to our system. On RH51, the TCP wrappers live in /usr/sbin/in.* so what's this in.sockd doing in /sbin? Being Naughty, that's what. I munged in.sockd through strings, and found some very interesting strings indeed. I quote: You are being logged , FUCK OFF , /bin/sh , Password: , backon . I doubt that this is part of an official RedHat release.

I quickly checked the other TCP wrappers, and found that RedHat's in.rshd is 11K, and the one on the HD was 200K. OK, 2 bogus wrappers. It seems that, looking at the file dates, this cracked wrapper came out the day after RH51 was released. Spooky, huh?

I noticed that these binaries, though dynamicically linked, used libc5, not libc6 which we have. Sure, libc5 exists, but nothing, and I mean nothing at all uses it. Pure background compatiblity code. After checking the other suspect binaries, they too used libc5. Thats where strings and grep (or a pager) gets used.

Now I'm getting bored of looking by hand, so lets narrow our search a little using find. Try everything in October of this year... I doubt our cracker was the patient sort - look at her mistakes so far - so she probably didn't get on before the beginning of the month. I don't claim to be a master of the find syntax, so I did this:

find / -xdev -ls | grep "Oct" | grep -v "19[89][0-7]" > octfiles.txt

One of the files reported by the find was /sbin/in.sockd. Interestingly enough, ps said that there was one unnamed process with a low (76) process id owned by uid=0, gid=26904. That group is unknown on campus here - whose is it? And how did this file get run so early so as to get that low a PID? In.sockd has that uid/gid pair... funky. It has to get called from the init scripts since this process appears on startup, with a consistently low PID. Grepping the rc.sysinit file for in.sockd, the last 2 lines of the file are this:

#Start Socket Deamon
exec in.sockd

The second daemon that was contaminated was rshd. About 10 times as big as the standard copy, it can't be up to anything but trouble. What does rsh mean here? RemoteSHell or RootShell? Your guess is as good as mine.

So far what we've found are compromised versions of ls, ps, rshd, in.sockd, and the party's just beginning. I suggest that once you're finished reading this, you do a web search for rootkit and see how many you can scrounge up and defeat. You have to know what to look for in order to be able to remove it.

While the log files had been all but wiped clean, the console still had some errors printed on it, quite a few after 0235h. One of these was a refusal to serve root access to / via nfs at 0246h. That coincided perfectly with the last access time to the NFS manpage. So our scriptkiddie found something neat, and she tried to mount this computer via NFS, but she didn't set it up properly. All crackers, I'd say, make mistakes. If they did everything perfectly we'd never notice them and there would be no problems. But it's the problems that arise from their flaws that cause us any amount of grief. So read your manuals. The more thorougly you know your system, the more likely you are to notice abnormalities.

One of the useful things (for stopping a cracker) about NFS is that if the server goes down, all the NFS clients with directories still mounted will hang. You'll have to 120-cycle the machine to get it back. Hmmm. This presents an interesting tool opportunity: write a script to detect an NFS hack, and if a remote machine gets in, ifconfig that interface off. Granted, that presents a possible denial-of-service if authorized users get cut off. But it's useful if you don't want your workstation getting compromised.

At this point I gave up. I learned what I'd set out to do - how to find the crap left behind by a cracker. Since the owner of this system had all her files on (removed) removable media there was no danger of them being in any way compromised. The ~janedoe directory was mounted off a Jaz disk which she took home at night, so I just dropped the CD into her drive and reinstalled. This is why you always keep user files on a separate partition, why you always keep backups and why it's a good plan to write down where to get the sources for things you downloaded, if you can't keep the original archives.

Now that we've accumulated enough evidence and we're merely spirited sluggers pulverizing an equine cadaver, it's time to consider the appropriate response. Similar to Meinel's you-can-get-punched and you-can-go-to-jail warnings in The Happy Hacker, I would suggest that a vicious retaliatory hack is not appropriate. In Canada, the RCMP does actually have their collective head out of the sand. I am not a lawyer, so don't do anything based on these words except find a lawyer of your own. With that out of the way, suffice it to say that we're big on property protection here. Aside from finding a lawyer of your own, my advice here is for you to call the national police, whoever they are. People like the RCMP, FBI, BKA, MI-5 and KGB probably don't mind a friendly phone call, especially if you're calling to see how you can become a better law-abiding citizen. Chances are, you'll get some really good tips, or at least some handy references. And of course you'll know someone who'll help you prosecute.

My communication with RCMP's Commercial Crimes unit (that includes theft of computing and/or network services) can be summarized as follows: E-mail has no expectation of privacy. You wish email was a secret, but wake up and realize that it's riskier than a postcard. As systems administrator, you can do anything you want with your computer - since it's your responsibility either because you own it or because you are its appointed custodian - so long as you warn the users first. So I can monitor each and every byte all of my users send or receive, since they've been warned verbally, electronically and in writing, of my intent to do so. My browse of the FBI's website shows similar things. But that was only browsing. Don't run afoul of provincial or state laws regulating the interception of electronic communication either.

NOTE: While I have attempted to make this reconstruction of events as accurate as possible, there's always a chance I might have misread a log entry, or have misinterpreted something. Further, this article is solely my opinion, and should not be read as the official position of my employer.

Appendix A: Programs you want in a crack-detection kit

• find, ps, ls, cp, rm, mv
• gdb, nm, strings, file, strip
• (GNU)tar, gzip, grep
• less / more
• vi / pico
• tcsh / bash / csh / sh
• mount

Appendix B: References WinFrame:
http://www.citrix.com/

RedHat 5.1:
http://www.redhat.com/
http://www.rootshell.com/ http://www.netspace.org/lsv-archive/bugtraq.html

About the filesystem:
McKusik, M.K., Joy, W.N., Leffler, S.J., Fabry, R.S., "A Fast File System for UNIX" Unix System Manager's Manual, Computer Systems Reseach Group, Berkeley. SMM-14 April 1986

LEA and Computer Crime:
http://www.rcmp-grc.gc.ca/html/cpu-cri.htm
http://www.fbi.gov/programs/compcrim.htm