Possible Defenses For DreamHost Security Breach


#1

Please use this thread for posting user-based solutions for the current DreamHost ‘Hacked-FTP-Password-Security-Breach’ situation ONLY. If you want to bitch about DreamHost please do it elsewhere.

This is a very serious breach, which appears to include hijacked passwords to FTP accounts, regardless of whether the password gets changed, and the resultant hacked index pages are forwarding on web site visitors to pages that may contain an exploit to Windows IE <6 and download malware key-logger onto their machine.

I will start off by trying to break down the problem, then posting possible defenses for different types of users: individual shared-server accounts and dedicated server accounts. These ‘solutions’ are my (and others) strategy and I make no guarantees whatsoever.

I only provide this information as a means to curtail the current damage for users until such time as DreamHost can solve the breach and provide a solution to users. (edit: I DO NOT work for DreamHost, and I am, frankly, not happy about the current situation. I just feel the need to do more than bitch. I can always bitch later. :wink: )

Step 1 is to stop the attacker’s access, Step 2 is to get rid of any compromised users, and Step 3 is to install a monitoring script to watch if further attacks happen.

[color=#CC0000]Please do not follow/try any of these recommendations if you do not understand them, or the means to complete them. Although these recommendations appear to have worked for me, and since we currently do not know the full extents or cause of the problem, they may not, in the end, keep it from happening again, even right away.[/color]


#2

uh…no thanks, I’ll pass on this one. :wink: I honestly don’t think I understand yet enough about what is happening to offer suggestions that I know will be productive and effective. :wink:

–rlparker


#3

(Step 1 of 3) First kill the means of access for the attackers; the FTP server.

(edit: As ‘rlparker’ noted ’ to “turn off” FTP for a user…just do it from the Control Panel -> Users -> Manage Users ->Edit user if you want to disable FTP for a given user or given users.’)

For dedicated server users

If you want to shutdown the whole FTP server process (proftpd), not just per user, the server is owned by nobody and you need to be root to kill it. So, you can ask DreamHost to do it for you, or you can do it, if DreamHost has given a user of yours ‘sudo’ privileges.

With sudo privileges, log in via SSH ($ is prompt):

  1. $ sudo top

  2. Use top and sort by command (type F then choose X) to get the PID # for ‘proftpd’ (DreamHost’s FTP server on port 21) then kill it (type k, then enter PID). Type q to quit top.

This is temporary as it may get started again by an automated DreamHost maintenance script and definitely at reboot; I didn’t want to mess with deeper, boot-time server settings at this time. proftpd has been the avenue of all attacks I have seen on my machine. SFTP uses port 22 (SSH) so you can still login and mess with files.

You could set a crontab to check for the server being up and kill it automatically, then delete the crontab once DreamHost says things are ‘fixed.’ Either way I don’t think the FTP server is necessary, if you use SSH/SCP, and should always remian off.


#4

I’d suggest reading the comments here first, especially some of the later ones: http://www.dreamhoststatus.com/2007/06/06/security-breach/#comments

Seems like there is a few things in play. The IFrame spam/keylogger trojan seems to be a reason why some users are being hacked repeatedly or despite changing all passwords. Personally, I would never use IE for any secure web-based interaction, especially web FTP or accessing DH’s webpanel.


#5

You may give the dreamhost users a false sense of security that if they follow directions from this post, then they don’t do any thing further, which may be a huge mistake, when in fact this may be malware and not a security breach.


#6

(Step 2 of 3) Next, find compromised users (bad users) and transfer any their directories over to a newly created user.

I am not sure if this is absolutely necessary, but an attacker can’t login using a user that doesn’t exist any more, regardless of password! If the DreamHost Web Panel is truly compromised then all of this is in vain. Let’s hope that isn’t the case.

1) Login to your server via SSH ($ is for prompt). (set the user to a shell user first in the Web Panel, if it is not)

2) $ last -ad

This shows the recent logins and converts their IPs over to host names. If a user is logged in (other than root) multiple times, probably from some weird host that your not from, and probably for only 00 minutes, then it is more than likely the automated attacker. The users associated with these accesses are always (as far as I can tell) through FTP (it will list ftpd####) next to the user. SSH logins will be listed as ‘pts/0’. Killing the FTP server should stop any ftpd logins.

3) Create a new user to replace the bad one. Immediately change the new user’s password to something else. DreamHost still sends out emails acknowledging new users have been created, with their password in plain text (WTF!??). Luckily, you can immediately change it; a process that does not email out the changed password.

4) Make sure you are in the user’s home directory, and find any files that where changed in the last week, skipping non-Web-accessible directories:

$ cd ~

$ find ./ -type f ! ( -path ‘cache/’ -or -path ‘logs/’ -or -path ‘Maildir/’ ) -mtime -7 -exec ls -alht {} ;

Fix any files that have been hacked. They should be files with ‘index’ in their names. You can also run this command, if you have uploaded a bunch of stuff recently:

$ find ./ -type f ! ( -path ‘cache/’ -or -path ‘logs/’ -or -path ‘Maildir/’ ) -name ‘index’ -mtime -7 -exec ls -alht {} ;

This only find index files changed within the past week, since known attacks have taken place.

5) Move or copy files from bad user to new user.

$ cd ~

$ tar -cvf mydomain.tar mydomain

$ chmod a(plus)r mydomain.tar
Note: replace (plus) with a plus sign. This forum keeps deleting the plus character!?

Make tar archives for every domain you want to transfer. Then make sure they are readable by the new user.

Log in as the new user under SSH.

$ cd ~

$ cp /home/baduser/mydomain.tar ./

$ tar -xvf mydomain.tar

Copy all needed web directories. Then de-tar them. The copied directories (and their contents) should now be owned by new user.

(edit: Logged in as the bad user, delete the tar archives you created. Optionally, first download them to your local machine for an off-server backup, then delete them on the server.)

6) In the Web Panel re-set the domains moved to the new user’s directory, or the domains will fail to resolve to the new location on the server (in the new user’s home diretory) after deleting the bad user.

7) Delete bad user from DreamHost Web Panel. This ‘should’ not erase the actual bad user directory, just delete the user login rights and information. You will not be able to login as this user again, so you should exit the current SSH session you may have open under that user. (edit: You may wish to remove the hacked directories, or the entire bad user directory contents before logging out, or deleting the bad user.)

NOTE: It may be beneficial, for attack tracking, to ‘strand’ the bad user. You can leave the bad user active, leave FTP on, delete everything but a couple of index files in the user’s directory, but switch your domains over to the new user. This way the domains will work, hopefully un-hacked, but you can help figure out if your machine is still being hacked by scanning with the ‘last -ad’ command to find any bad FTP logins.


#7

I work on a Mac and do not use Microsoft Internet Explorer. I use FireFox. I changed ALL of my passwords, then one day later, the same FTP user was being logged in again and hacking away. A key logger was not part of the problem on my local machine.

This is a security breach on DreamHost’s system, and at a level that user’s, even with root (sudo) access to their servers can not block. The key-logger issues are secondary, a symptom. (edit: Though key loggers may have started the breach on a DreamHost employee’s machine. It’s really an unknown right now. I have followed every post on the breach’s status page www.dreamhoststatus.com, but there is not much for solutions there, mostly just whining and arguing.)

I am merely trying to helping people keep their sites up by sharing what I have done that has worked.


#8

@macinarizona

[quote]when in fact this may be malware and not a security breach.

[/quote]

There does seem to be malware in the mix, but I can tell you with great certainty that at least one account was compromised from DH’s side. This account is only accessed via SSH & SFTP and only from an OSX machine. The passwords for that account were not obtained from “sniffing” or from a keylogger.


#9

That is certainly part of why I think this thread is premature. My feeling is that the forensic analysis is not complete, and such suggestions for “defense” (while some of them might well be good ideas) are based on incomplete information.

I prefer to wait until all the facts are in, and more is known about the nature of the compromise and the vectors used are authoritatively identified.

There are some very smart people about, and their ideas may have some validity, but, for now, I’m reading mostly speculation and assorted FUD than I am real facts, and prefer to take my information from DH or people whose credentials I know and trust.

–rlparker


#10

There is no reason at all to “contact support” to “turn off” FTP for a user…just do it from the Control Panel -> Users -> Manage Users ->Edit user if you want to disable FTP for a given user or given users. :wink:

–rlparker


#11

(Step 3 of 3) Set up a monitoring script that emails you results, so you can see logins and changed files

1)Make a shell script in the home directory of users you want to monitor and name it something (I named mine ‘monitor-access.sh’). Add the following commands to it:

[code]#!/bin/bash

echo
echo “## Files Modified within last 140 Min. ###”
echo
find /home/myuser/ -type f ! ( -path ‘logs/’ -or -path ‘Maildir/’ ) -mmin -140 -exec ls -alht {} ;
echo
echo
echo “### Recent Logins and Associated Hosts ###”
echo
last -ad
echo
echo
[/code]Replace /home/myuser/ with whatever your user path is. you can add more directories that you don’t want to search to the ! expression like

\! \( -path ‘*cache/*’ -or -path ‘*logs/*’ -or -path ‘*Maildir/*’ \) You can run this script manually by logging in via SSH with the command ($ is prompt):

$ cd ~

$ ./monitor-access.sh

2)Set a crontab to run the script:

Go into each user’s home directory you want to monitor

$ crontab -e

This brings up a new (or your current) crontab in your current editor. Or you can also add the cron scheduling commands in a text file, then use the following to load the text file into the crontab:

$ crontab my-uploaded_cron-text-file.txt

The cron commands are as follows (Though you can set the interval to what you like, the shell script works, with some overlap with this interval. So, if you change the interval, you need to change the ‘-mmin -#’ to have a little overlap with your cron interval):

0 */2 * * * ~/monitor-access.sh

This runs the monitoring script every two hours and will email you the results to the address user@server.dreamhost.com, where user is your user and server is the name of the server you are on. If you don’t get mail, for some reason, at that address, or you are monitoring multiple user accounts, you can have the server mail to a particular email address using:

0 */2 * * * ~/monitor-access.sh | /usr/bin/mail -s “Cron monitor-access.sh for user” my@emailaddress.com >/dev/null 2>&1

Switch the word user with the name of the user running the crontab. This will email the results to any address (my@emailaddress.com) using the ‘mail’ program, then delete any results (>/dev/null 2>&1) so that cron doesn’t also mail out to user@server.dreamhost.com.


#12

[quote]That is certainly part of why I think this thread is premature. My feeling is that the forensic analysis is not complete, and such suggestions for “defense” (while some of them might well be good ideas) are based on incomplete information.

I prefer to wait until all the facts are in, and more is known about the nature of the compromise and the vectors used are authoritatively identified.

There are some very smart people about, and their ideas may have some validity, but, for now, I’m reading mostly speculation and assorted FUD than I am real facts, and prefer to take my information from DH or people whose credentials I know and trust.

–rlparker[/quote]
I appreciate your desire for a solution from DreamHost. I too wish they would come forward with either workarounds or solutions, but they haven’t in DAYS. I could not wait days for them to solve the problems, while I sit around and have my clients’ sites hacked repeatedly. Some of the users on my dedicated server machine are very upset and I had to find a solution immediately.

These steps have worked for me, and other than possibly some of the dedicated server user sudo commands, are not against the DreamHost terms of use (it is unclear what you can or can not do as a sudo user, if it does not ‘harm’ the operation of the machine). I am paying $1800/year for my dedicated server, and while I expect immediate service at that price, I also expect the ability to fix ‘what I can, when I can’ without having to wait for support.

For the normal shared hosting user, these recommendations are not damaging, or even harmful, changes.

You are certainly welcome to wait around while your sites are being hacked. After two documented attacks, I applied these recommendations and mine are no longer being hacked.

It is by all means your choice. In the meantime, thanks for all your help.


#13

You make some very valid points, and personally I like some of your suggested recommendations. I probably was “less than clear” about that (though I did say some of them were “good ideas” in an earlier post).

I think it is true that, as a “dedicated server” customer with (now) demonstrated *nix skills, you might be underestimating the ability of many typical “shared hosting” customers ability to badly bork implementing some of what you suggest (in spite of your clear instructions!). A quick tour through the posts in these forums make that possibility pretty obvious.

The fact that these suggestions worked for you is valuable, and it is admirable that you took the time and effort to prepare these posts - note that my first response(s) to this thread were made before your actual suggestions, when it appeared possible that the thread might devolve into a collection of speculation and questionable advice. :wink:

I’ve been monitoring this situation since May 23rd, and fortunately, my sites have not been hacked, so I have not been faced with the urgency of crisis that you have; I’m glad that your countermeasures are proving to be effective and am hopeful that your sharing them will be of use to others. Again, thanks for taking the time to do that.

As for the “thanks for all the help” comment; I’m more than happy to provide help (as I often do on these forums) but choose to know more about the problem before I offer suggestions; I apologize profusely, now that I see the quality of your posts, for responding in a dismissive way to your initial post. :slight_smile:

I just don’t want to see a whole slew of unjustified and questionable “fixes” and rhetoric infest the forum, as they have the comment section of the blog. PAX.

–rlparker


#14

[quote]As for the “thanks for all the help” comment; I’m more than happy to provide help (as I often do on these forums) but choose to know more about the problem before I offer suggestions; I apologize profusely, now that I see the quality of your posts, for responding in a dismissive way to your initial post.

–rlparker[/quote]
It was inappropriate of me to imply you may not be helping. It is obvious that you do. A healthy amount of skepticism a good thing :slight_smile: , and you’re right to warn folks about just blindly doing what someone suggests to fix something. Impatience is not a virtue. No one should be doing any of these recommendations unless they understand them and are capable of executing them. Obviously, not everyone is capable, but they may try anyways (I know I would). So your warnings are certainly justified. I will edit the first post accordingly.

Thank you.


#15

Good looking functional script.

If I were on a dedicated host, I would create something like your script. Specifically, I would probably run tripwire.

If I put your script in for my small number of users, I agree it would not present much load to the system.

I am on a shared host. If 200 users run this script, would this still be true?

If the consensus is our machines can handle it, it sounds like a good idea, regardless of the current issue. I am sure it will happen again someday.

Regards,
Rudy


#16

Personally I only had 6 accounts reported as compromised by DH out of 240 or so, I don’t think it’ll kill the server if I run the find script on those 6 a few times a day. It’s already paid off, made it easier for me to check for modified files on those accounts today. I’ve been watching the output of last myself and I think that’s how DH figured out what accounts are being accessed by the attackers, it’s pretty obvious when you see 2-3 different accounts logged into multiple times in a span of 5 minutes all from one IP address.

I’m glad for this info, I’ve already been doing what I can to minimize the impact because it has become very apparent that following DH’s recommended advice is pointless so far. It’s a bit depressing to change the passwords and see the attackers continue to login. Personally I suspect there’s a zero-day exploit out there for proftpd that’s at least part of the problem. If not then there’s something seriously wrong because there’s far too many people reporting continued logins after password changes that couldn’t have been infected with the keylogger in question (those using Linux, MacOSX, Firefox, etc.) I think the fact that disabling FTP on the accounts stops the attackers is another point against the mass keylogger theory – the attackers would see the changes and just switch to SFTP instead. (SFTP will continue to work on a shell account with FTP disabled.)