In the event of a meltdown


#1

I’ve been tasked by my boss to draft a soup-to-nuts site restoration plan, in the event our website is completely compromised by a third party.

Operating under the assumption that they have changed all our passwords killed all cron jobs, and basically acted as if they knew what they were doing, what’s the most efficient and quick way to get the “new” site blown away and our password reset so we can re-upload the site?

Thanks!


#2

This is part what should be a comprehensive disaster recovery plan. The way I handle this is that I regularly sync my files and databases to multiple locations. And I keep several versions of the file (e.g. 2 hours ago, last night, 2 nights ago, last week… etc). That way not only do I have a backup, but I have a point in time to recover to.

I download and sync my files to my local computer as well as 2 other servers. I do this automatically every 2 - 12 hours depending on the site. So in case that my site gets compromised I can easily take one of my backups, review it and then re-upload it to the site.

I use a combination of tools for this and your mileage may vary but they are – cron, mysql administrator, mysql, rsnapshot, rsync, and ssh. – pm me if you need more assistance.

– oh yeah and as for resetting your password, you’d do that just by contacting DH. And I would do this before I try fixing anything… I would also start in a new account or ask DH to erase your account and start over (provided you have a backup of course). This eliminates any chance that the attacker could have left a back door or something else that you might miss like SSH-keys, or hidden web scripts.


#3

Yep, a disaster recovery plan. My part is the website.

The assumption that an attacker has changed the FTP password and killed all cron jobs (which I would expect of an attacker-with-a-purpose) pretty much means that nothing depending on a cron job or requires I log into my FTP account first is gonna fly.

So, as far as I can tell, that leaves “write an email to support@dreamhost.com”. Presumably in that email, I have to convince them sufficiently that I’m the original accountholder, that I’ve been pwned, and that our site needs to be nuked.

On our end, a human being can restore the site to skeleton shape within five minutes and have it fully functional within another 10. So, we’re cool there.

We also now have our server monitor tools monitoring the change-data for several of our top-level files, which means that not only will we know if we get pwned withing a few minutes, but we also know what our typical uptime is (as of a few days ago).


#4

Your assumption is both correct and incorrect… an attacker is more likely to leave things as is and only change things to their benefit. If they were to kill all that stuff, then you would discover them easily and their attack would be quickly killed. But then again, they might do that just to get a quick batch of links or just to deface your site…

with that in mind… the cron job and stuff I’m talking about isn’t on the hosting account… as i mentioned. I run these from other servers and essentially copy and backup all the stuff in the DH account to an offsite location…

The logistics of how to get DH to know you are the account holder are simple. You are the one paying the bills and thus that is your ultimate proof that you are who you say you are.

So the scripts, cron jobs, and backups would reside outside of DH. So if an attacker messes up with your account, the offsite files that you’ve been backing up should be out of the attackers reach because presumably they’d be in a secure local server to your office or workplace…

Do I make more sense now?


#5

"Do I make more sense now? "

Yes, running a cron job on another site makes sense. As I mentioned, we now have a tool monitoring all of our servers in-house, and it’s a fairly simple matter for it to also monitor the status of several files on our website, by SSHing over and checking the files on a regular basis. It’s not an ACTUAL FIM, but without a DH-based FIM, it’s pretty decent.

One of my operating parameters was that the attacker has completely compromised the website and is probably bragging about it – not a subtle attack at all. This would be particularly awkward for us, reputation-wise – we could lose customers pretty much instantly if that happened. Were I to be told about this, I would be expected to wipe and restore the site in minutes, if possible.

However, it is now my understanding that a login to the Control Panel from any IP other than the usual one (even with the proper credentials) does not allow access to the Control Panel until the registered email address has confirmed back that this IP access is okay.

So, that helps a great deal (and makes good sense anyway).

In order for someone to compromise our website through a credentialed attack, they would need not only our credentials, but to be able to spoof our IP.

As far as what DH considers authentication, using the Web Panel is probably good enough, considering the above information.


#6

Sorry Edward… I’m not sure what you’re looking for. I posted a great solution that I use in a number of sites to complete a DR plan. if you have a monitoring system like Nagios or Zabbix or something else that is flexible, you can even have this system trigger the DR automatically and notify you so you can complete any manual steps in a timely fashion.

say that your site is example.com and everything is fine… 8 am Monday morning…
the system I suggested is running fine and there is a recent snapshot or backup (within an hour of 8 am). that is available.
your site gets compromised at 8:30 am…
you lock out access to DH and get your account reset or gain access.
establish a “down for unexpected maintenance banner” while you…
you restore the recent backup files…
After a short while, site is back up and running with minimal data loss and business disruption.

All the recovery and backup scripts/jobs/files reside outside of DH so even if you had total compromise of the DH account you would have everything offsite to restore to a clean state. Depending on the size of your site and your internet connection speed, you could have your site up and running in < 15 - 20 min.

"However, it is now my understanding that a login to the Control Panel from any IP other than the usual one (even with the proper credentials) does not allow access to the Control Panel until the registered email address has confirmed back that this IP access is okay."
Where did you get this from? – I regularly log into my control panel from multiple ips without a problem… I’ve never had to confirm access, but I guess I could be using a different plan. (I’m on a PS plan).

I’d be happy to discuss this via PM if I’m not being clear on something. But if you understand what I’m suggesting and it isn’t what you’re looking for then that’s fine too, and I wish you good luck.

Oscar

ps. one more thing… you say that you have the monitoring system and it can check for files and stuff… but when I talk about an offsite system, I’m refering to having the system synchronize and backup your files… offline, not just monitoring them. I just re-read your response and wanted to clarify that…


#7

“All the recovery and backup scripts/jobs/files reside outside of DH so even if you had total compromise of the DH account you would have everything offsite to restore to a clean state. Depending on the size of your site and your internet connection speed, you could have your site up and running in < 15 - 20 min.”

Yep, that’s pretty much where we’re at right now.

We have a ZenOss system that monitors in pretty-much-live-time all our internal servers. Apparently, it was child’s play to add the website to it, so now I have a functioning equivalent of a FIM.

“Where did you get this from? – I regularly log into my control panel from multiple ips without a problem… I’ve never had to confirm access, but I guess I could be using a different plan. (I’m on a PS plan).”

Heh – one of the DH support guys told me this.

So, I tested it by logging in from a different IP, which worked great.

I told him about that, and he checked and wrote back – apparently, it’s a setting and the setting is OFF by default.

After he set it, I tested it again, and we were locked down.

And yeah, we’re backed up every 24 hours completely.


#8

I have found that you have to craft your question to the support guys carefully to get the right answer, just like in the forums I guess… but it seems like you have everything under control… So what do you still need help with?

To me, a proper DR plan… especially one that is to be used in a dire situation as you described needs multiple backups per 24 hour period. Depending on activity, these could vary from hourly to every 3 - 4 hours… 24 hour backups are not enough for this unless you’re willing to loose 24 hours of traffic data, content changes and other things you might be doing on your site.

Anyway. pm me or reply if you still need help but I’m not sure what else I can tell you…


#9

It’s okay – it’s good to have another perspective on it. Thanks muchly!

Our site is currently flat HTML, no interactivity. So, we always have the latest copy in an inside-the-firewall location.