Hacked! How to trace?


#1

This morning I got a notice from Google that they were going to drop my pages due to hidden references to viagra.

I investigated and found a big blob of html that had been added to at least 3 specific files across my account - possibly more.

Needless to say, I’ve already changed my password - but how can I go further in figuring out how this happened?

Thanks.


#2

If you haven’t already done so, switch to using SFTP instead of FTP, and disable FTP access to your account.

si-blog
Max discount on any plan with promocode SCJESSEYTOTAL


#3

You might want to start by reviewing your logins to see if your site was accessed by someone other than you.

From the shell:

last -100 username

will produce a listing of the last 100 logins, and you can check to see if all those are from IP addresses you recognize, or are at times you actually accessed the site.

–rlparker


#4

What is the URL of your site, by the way?

si-blog
Max discount on any plan with promocode SCJESSEYTOTAL


#5

Make sure all scripts are secure and up-to-date.

That’s cool that Google gave you a heads up instead of just dropping you and waiting for you to figure it out on your own.


:stuck_out_tongue: Maximum savings promo code: MaxSavingsAtDH


#6

hi there,

Given that I am dealing with an exploit on my own site, would disabling FTP halt any current hacks or exploits, or is this just a preventative for the future?


#7

I only mentioned it because the one and only time my site was compromised it turned out to be because my FTP details were “sniffed”. Switching to SFTP (I use WinSCP now) eliminated that particular problem.

si-blog
Max discount on any plan with promocode SCJESSEYTOTAL


#8

This only helps prevent future hacks, but only after you change your password (obvious, yes).

There tend to be two ways sites get hacked:

  1. Someone got your password (sniffed, guessed, whatever)
  2. Script exploit because you’re running unsecure software

-Scott


#9

Thanks for the tip about last. I went through the records and indeed there were a few strange IPs from Latin America and elsewhere in the FTP logs. Fun.

Now how can I figure out what damage was caused?


#10

You are welcome. It’s really hard to “guess” what the best approach would be. Obviously it would be nice if you could tell what was done during those logins, but that may not be so trivial, especially if the perpetrator “touched” the files he manipulated (obfuscating time stamps).

If your user (the one that was logged in from the suspect IP addresses) had shell access, you may be able to tell what was done by scrolling through the bash history (to see what commands the perpetrator executed).

He may also have just uploaded stuff (possibly overwriting existing files), which is harder to tell without full access to the FTP logs (and I don’t believe you have access to them - that may require DH support’s help).

The approach I would take is to assume the worst, and accept that you may not be able to (quickly or conveniently) tell exactly what was done, and proceed as follows:

  1. Change Password (which I assume you have done)

  2. Disable FTP access to the server for your user (enable SFTP/SSH only)

  3. Erase everything

  4. Reload directories from known good backups

  5. Update all scripts you are running to their latest versions.

I don’t know if you can rely on DH “.snapshot” backups to be “known good” (depends upon when the perpetrator found his way into your account), so I would use only stuff I had stored “off-site” for the reload.

That process should get you “back in business” as soon as possible, but still leaves the possibility that one of your scripts was exploited. You can investigate this a little further with a careful review of your access logs to see what, if any, “strange stuff” was passed via http requests, and a thorough Google search for exploits related to any software you are running.

–rlparker


#11

I guess I can do ls -R | grep 2008 to see all the files that were changed so far.

Is it possible in unix to change the time/date stamp to something in the past?


#12

I think the “off site” backups you have at home is in the same reliability boat as what’s in the .snapshot. You would have to know when the site was compromised before you can pick a safe backup to restore. Snapshots go back a couple of weeks, so if you’re confident the compromise happened only within the past week, then snapshot or a local backup would work.

I’d just hate to have to recover from such a compromise by picking through backups. Hopefully you have a good reliable set.

-Scott


#13

Yes, it is, so searching for timestamps won’t be reliable, but it may help.

-Scott


#14

Yeah, very true … I typically have a “deeper nest” of my “off site back-ups” than the 2 week window the .snapshot provides, but the key is certainly to identify the “last known good” set of files. :wink:

Me too, on both points! :slight_smile:

–rlparker


#15

As Scott replied, “touch” can do that (see"man touch" or “touch --help” for the “-t” option).

Also, sometimes a perpetrator might just touch all files to make them current time further obfuscating the real activity - either way it makes timestamps less than completely reliable.

The truly devious will touch files in such a way to make it appear that the “new” stuff is actually part of a “known good” set of files … that sucks. :frowning:

–rlparker


#16

Wow this really blows. :frowning:

Two lessons:

  1. Don’t log in from asia using ftp.

  2. Backup more regularly.


#17

I tried this just to see what would show up. I used ssh to go into two different domains owned by different users and ran “last -100 username” for both of them. They both showed the following output:

wtmp begins Tue Jan 1 06:42:54 2008
wtmp begins Tue Jan 1 06:42:54 2008

This is not what I expected to see. Any thoughts on what is being returned?

[quote]You might want to start by reviewing your logins to see if your site was accessed by someone other than you.

From the shell:

last -100 username

will produce a listing of the last 100 logins, and you can check to see if all those are from IP addresses you recognize, or are at times you actually accessed the site.

–rlparker[/quote]


#18

The lastlog does get reset from time to time. From what I usually see, that data is gone if it’s more than a few weeks old. Lastlog is a rotated log that eventually rolls off.

Seeing how your wtmp begins at the new year, that implies that nothing’s happened on your account in the past 10 days. I hope that instead of “username,” that you typed in your real username.

-Scott


#19

What Scott said! For instance, on one of my user accounts, I only have 5 entries presently available (all very recent)… so it won’t always be useful if the log has been “restarted”.

I wonder if DH support has the “older” log info available for forensic purposes? :wink:

–rlparker


#20

looks like there was less activity for some users that i realized. that’s a cool command though. thanks for sharing it.