Just Went Out for Bread, or the Story of One Hack
A freelancer hired to configure an email server stumbles upon an active intruder's screen session, discovers replaced SSH binaries, a Linux rootkit, and has to rescue the system from cascading segmentation faults.
It all started when someone approached me (as a freelancer) for help and asked me to configure exim4 so that their email newsletter wouldn't end up in spam. They even thoughtfully sent me a link to an excellent article.
A couple hours of work including DNS updates, but things didn't go as planned. After logging in as root, I launched my beloved screen out of habit with the command screen -x and witnessed a most curious spectacle in everyone's favorite /dev/shm directory. The attacker hadn't bothered to close the screen session, or was perhaps still working in it. And so the quest begins:
The first thing I did was review what the attacker had been up to:
wget http://ravenul.zzl.org/it/noi/up/8.txt mv 8.txt list.txt php lol.php php lol.php netstat -an | grep :22 w rm -rf list.txt w rm -rf .x netstat -an | grep :22
Apparently they were sending spam and running some file ".x" (or was it a directory?), and also checking SSH connections. There was also an archive with a PHP script lol.php, which I unfortunately forgot to save.
The output of last and who commands showed nothing supernatural — no root sessions for a month, which the server owner confirmed. However...
$ lsof -ni | grep ssh
...showed an established connection with IP 172.190.125.14, which I immediately killed.
I noticed something about /usr/sbin/sshd:
$ ls -la /usr/sbin/sshd -rwxr-xr-x 1 root root 320724 Oct 11 23:29 /usr/sbin/sshd
Next to sshd sat sshd0:
$ ls -la /usr/sbin/sshd0 -rwxr-xr-x 1 root root 757356 Jul 31 2010 /usr/sbin/sshd0
Deleting the file didn't work:
$ rm -f /usr/sbin/sshd rm: cannot remove '/usr/sbin/sshd': Operation not permitted
Let's dig deeper:
$ lsattr /usr/sbin/sshd -u--ia------------- /usr/sbin/sshd $ chattr -aui /usr/sbin/sshd $ rm /usr/sbin/sshd $ lsattr /usr/bin/* | grep -v -- '-------------------' -u--ia------------- /usr/bin/ssh $ chattr -aui /usr/bin/ssh $ rm /usr/bin/ssh
I reinstalled openssh-server and openssh-client. Everything seemed fine, no more threats, nothing else suspicious found. I decided to also update the system, and the tzdata was old (hello, Medvedev and the timezone reform!). I checked /etc/apt/sources.list and /etc/apt/sources.d. All files were in order, no suspicious entries, dates unchanged for a year. After apt-get update, I applied all security updates to Debian Lenny, including a new kernel. Well then. Time to reboot. Just in case, I requested KVM access (which turned out to be a wise decision) and started waiting.
The next day, KVM access was provided. I typed "reboot" and then — bam: dozens of segmentation faults. Hair turning grey, hands shaking. I think many can imagine my situation. As they say, "if it works — don't TOUCH it!" but after discovering the intrusion, I had to apply updates and reboot.
In short, I pulled myself together, began investigating what was wrong, and booted into single user mode. The mount command caused a segmentation fault every time it was called, even without parameters. The filesystem was read-only, nothing could be done. /etc/fstab was fine, df also worked. The date command also segfaulted for some reason. I ran a disk check (software raid1) fsck.ext3 /dev/md0 — everything was fine, no anomalies. What was going on? I started thinking that I had crashed the system myself, since I had updated the tzdata package, which is directly related to time. And then the DSL connection with my ISP dropped... I rebooted the modem — connection came back up, great!
The server owner was furious, as the server had been down for several hours, and decided to submit a ticket to "Infobox" support. Meanwhile, I'm stressed out, continuing to dig through the system. The most sensible solution seemed to be to reboot the machine and boot from a liveusb so the disk would be RW, then figure it out from there. I started debugging mount with whatever methods were available at the moment. gdb wasn't installed, only ldd was available, which showed nothing serious, and export LD_DEBUG=all, which also revealed nothing supernatural. The segfault simply started after initialization of all libraries. Then KVM told me it was disconnected. Right, support arrived. I walked away from the laptop and continued thinking...
While standing outside and breathing fresh air, a very educated cockroach ran into my head and said, "What if the files causing the segfault are replaced?" Said and done. I waited to hear what the client would say about the support ticket. A few minutes later, he forwarded me the support response:
The partition table is damaged, and it cannot be restored by express methods.
If you wish, we can engage our system administrators (the cost of work is 870 rubles per hour) for restoration.
Alternatively, you can do it yourself. In that case, we recommend using Gpart (http://packages.debian.org/ru/sid/gpart)
Holy cow, I thought... I told the client that this couldn't be, since fsck had checked the disk and found no filesystem violations. The client wrote a response to support, and meanwhile KVM access returned, where I saw the same futile attempts to call mount, hdparm (which wasn't installed), and work with fdisk.
The latter displayed nothing other than:
$ fdisk -l Disk /dev/sda: 160.0 GB, 160041885696 bytes 255 heads, 63 sectors/track, 19457 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk identifier: 0x000f0571 Device Boot Start End Blocks Id System /dev/sda1 1 18480 148440568+ fd Linux raid autodetect /dev/sda2 18481 19457 7847752+ fd Linux raid autodetect Disk /dev/sdb: 160.0 GB, 160041885696 bytes 255 heads, 63 sectors/track, 19457 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk identifier: 0x00000000 Device Boot Start End Blocks Id System /dev/sdb1 * 1 18480 148440568+ fd Linux raid autodetect /dev/sdb2 18481 19457 7847752+ fd Linux raid autodetect Disk /dev/md0: 152.0 GB, 152003018752 bytes 2 heads, 4 sectors/track, 37110112 cylinders Units = cylinders of 8 * 512 = 4096 bytes Disk identifier: 0x00000000 Disk /dev/md0 doesn't contain a valid partition table Disk /dev/md1: 8036 MB, 8036024320 bytes 2 heads, 4 sectors/track, 1961920 cylinders Units = cylinders of 8 * 512 = 4096 bytes Disk identifier: 0x00000000 Disk /dev/md1 doesn't contain a valid partition table
Based on those last lines "Disk /dev/md0 doesn't contain a valid partition table," support concluded that the problem was with the partition table. Indeed, how didn't I think of that earlier! After all, fdisk has never shown partition tables for software RAID. I relayed all my thoughts to the client and began developing the cockroach's devious plan. I can only imagine how the support saga would have ended and how long it would have taken had the client agreed to their help. And the cost is easy to calculate.
I looked at the modification date of /bin/mount — it showed the time of the last server boot. I rebooted, checked the date again — time of the last server boot. Strange. So something was modifying this file during boot, and that "something" needed to be dealt with.
/tmp was read-only. To upload a file to the server, I needed a writable filesystem. I remembered /dev/shm. I brought up the network interface, assigned an IP, and downloaded the mount deb package for Lenny. Unpacked it, ran it — voila! It works! I remounted the filesystem, now it's RW. Things are moving!
I checked the files in /bin/ and saw the following picture:
$ ls -latr /bin -rwxr-xr-x 1 root root 96408 Nov 15 18:11 vdir -rwxr-xr-x 1 root root 30896 Nov 15 18:11 pwd -rwxr-xr-x 1 root root 30712 Nov 15 18:11 ping6 -rwxr-xr-x 1 root root 24252 Nov 15 18:11 nc.traditional -rwxr-xr-x 1 root root 8612 Nov 15 18:11 mountpoint -rwxr-xr-x 1 root root 68208 Nov 15 18:11 mount -rwxr-xr-x 1 root root 32244 Nov 15 18:11 mknod -rwxr-xr-x 1 root root 39144 Nov 15 18:11 loadkeys -rwxr-xr-x 1 root root 17244 Nov 15 18:11 kill -rwxr-xr-x 1 root root 9764 Nov 15 18:11 fgconsole -rwxr-xr-x 1 root root 26216 Nov 15 18:11 false -rwxr-xr-x 1 root root 8524 Nov 15 18:11 dmesg -rwxr-xr-x 1 root root 96408 Nov 15 18:11 dir -rwxr-xr-x 1 root root 51988 Nov 15 18:11 dd -rwxr-xr-x 1 root root 59148 Nov 15 18:11 date -rwxr-xr-x 1 root root 49440 Nov 15 18:11 chgrp -rwxr-xr-x 1 root root 30956 Nov 15 18:11 cat -rwxr-xr-x 1 root root 12252 Nov 15 18:11 bzip2recover
The modification dates of these files changed every 3 minutes and 10 seconds. I started examining crontabs, found nothing. Catching which process was modifying files with lsof didn't work. I ran ps auxww and saw a certain process cat /sys/class/net/lo/operstate hanging around.
I downloaded a package with the kill utility, renamed /bin/cat to /bin/cat_ and killed the process. The files stopped being modified. Victory. Now all that remained was to replace all modified files with originals. I downloaded the necessary packages and installed them via dpkg -i *.deb, having first checked the creation date of dpkg itself. After all the replacements, with fingers crossed, I typed reboot and watched the KVM window. Boot was successful, the site was working. Then I scanned the infected files I had copied using ClamAV and discovered Linux.RST.B-1 FOUND. Who said there are no Linux viruses? By the way, it's a virus from 2001...
Scanning sshd and ssh yielded nothing definitive. Apparently they were simply modified ssh and sshd binaries. The first most likely sent the login and password upon successful server connection, the second most likely let anyone into the server with a specific password. I don't have the energy to dig into these files right now, but those interested can download them and investigate: zalil.ru/32063611
P.S. If anything in the commands is off, I apologize — many of them were written from memory. The desire to configure exim4 also faded. I haven't asked for payment yet. And for what? I didn't even complete the main task =)
P.P.S. Hello to Infobox!
