All file permissions changed to 755 -- not by me


#1

I noticed all my files on mothman, webpage files as well as offline files in other directory trees, have been chmod’ed 755. Not by me. I sent dreamhost a “ticket” about that. They replied with some boilerplate “what to do if your site is hacked” material. I replied back, telling them it looks to me like an inside mistake, likely dreamhost restoring from backup tapes (or other medium) and messing up permissions in the process. Maybe they were swapping disks, had a crash, or something. I asked them to review their activities on mothman, let me know what actually happened, and what steps they were taking to make sure it doesn’t happen again.

Anybody else on mothman have this happen? Or can you check and tell me it didn’t happen to you?

I’m now seeing that the likely incident is…

http://www.dreamhoststatus.com/2012/01/22/web-vps-host-homie-vserver233-moved-to-new-hardware-and-restoring-from-backups/

http://tinyurl.com/75wefwj

…and not any dreamed-up dreamhost hacker. So, again, I’d like to publicly ask dreamhost to (a)confirm this is what happened, (b)describe exactly what backup/restore mistake occurred, and ©discuss what they’re doing to make sure future backup/restores are performed correctly. And I suppose it would help prompt dreamhost to address this issue if other people who’ve experienced this problem follow up.


#2

Why do you think that the incident with homie-vserver233 for VPS’s has anything to do with shared server mothman?

Have you checked to find out if your site was hacked? Have you looked in .htaccess or run a ‘find’ command, as documented on this page: http://wiki.dreamhost.com/Troubleshooting_Hacked_Sites

The hacks that have been going around apparently only manifest themselves as a conditional redirect when the user is refereed to a site from a search engine such as google. This makes the hack live longer because it’s very rare that the site owner or regular users would use search results to transfer to the site, they would more than likely access the site directly and never see the hack in action.


#3

Am I just imagining it, or do I detect an interesting evolution in “recommended practice”…

In the old days, newbie customers were (I think) encouraged to create users with SFTP but not SHELL access … especially if they didn’t know what a “shell” is …

but now everyone is being encouraged to do shell commands to periodically check their websites,

so we should be creating all our users with SHELL access. Is this correct?

~Tom


#4

using a shell is the most efficient, and sometimes the only, way of doing things. it’s nothing more than a command line interface, although you could run a gui over an ssh connection. i don’t know if all of the utilities that you might want to use have guis available. it’s also easy to screw things up, so it helps to read up before you do anything.


#5

Yes (to lakerat), as far as I can tell, my site/shell account haven’t been hacked. So, that aside, the problem looked backup/restore related to me, and the incident I cited seemed the best available candidate (you know, kind of like the republican party’s situation:). But in that case, I’d have expected other people to report the same experience. And nobody has, so far. So I’m less convinced of my initial guess, but can’t dream up any alternative. If you assume no hack, how do you otherwise account for all permissions, on every single file everywhere (web and “offline” files), changed to 755? Note that that’s not group/world writable, and at best lets people read previously-600 files (of which I have many). And no files seem missing or “damaged” in any way, as far as I can tell (by diff’ing them against tarball backups that I periodically make and download to my soho lan). So my best (and only available) guess is still that it’s gotta be some kind of dreamhost foul-up of some kind. And, by the way, this time they haven’t replied to my email at all, though on several previous occasions they’ve been very prompt, courteous and helpful about replying.

And, vis-a-vis bobcat’s shell remarks, I’m a professional Unix C programmer (for decades, see my resume at http://www.forkosh.com/resume.html) and probably didn’t “screw things up” myself, although, of course, anything’s possible (i.e., during those decades I’ve indeed taken ample advantage of many opportunities to screw things up:).


#6

Check you .htaccess file and see if there is anything in it that forwards users to a domain unknown to you if the referrer is google et al…


#7

[quote=“LakeRat, post:6, topic:57034”]
Check you .htaccess file and see if there is anything in it that forwards users to a domain unknown to you if the referrer is google et al… [/quote]

Nope, it’s fine. Byte-for-byte identical to tarballed backup (which I’d already checked with a script that diff’ed everything, and found only differences I made myself and expected to find). But thanks for trying. Why can’t it be a dreamhost foul-up?


#8

It could be. On the surface the permissions changed to 755 sounds like one of the things that’s been happening lately in the rash of hacked sites. Also as a regular in this forum, I can’t recall a case where file permissions were changed magically by a dreamhost mistake.

The doorway in those cases is usually found to be some insecure theme or plugin someone installed because they think it’s perfect for their needs, but came from an untrusted source.

Support is usually pretty good about the answers they give, but they like anyone can make a mistake.

Also the last time shared server mothman showed up on dreamhoststatus was Sept 2nd, 2011 http://www.dreamhoststatus.com/2011/09/02/shared-server-mothman-being-transferred-to-new-hardware/


#9

All the mod times are the same? Or are they all original mod times? If someone chmoded everything, then all the mod times should be similar/same.

Assuming they are, find out who was logged in at that time with the last command. Failing that, check for that time in the logs to be sure nothing malicious was happening at that time.

Failing that, then it likely is a DH issue and should be addressed. In light of the lapses over the past few months / year, it would help if DH became a bit hyper-vigilant for a while re: security.


#10

Thanks, “bobocat” (sorry for calling you bobcat earlier:), that’s a good idea. I’ve already chmod’ed lots of stuff back, but enough remains so that ls -alct|less makes it clear everything happened 2012-02-15 23:00 (late this past Wednesday, way past lakerat’s 9/2/11), at which time it looks like only “omalonzo” and I were logged on. Kind of suggests I did it myself, but what easy-to-make typo would do that? I can understand an easy typo affecting one directory, e.g., you want rm abc.* but accidentally put a space between the dot and asterisk. But I can’t quite see an unnoticed typo sufficient to affect every directory and subdirectory above the pwd. Too many extra chars required to go unnoticed. But I’d actually feel much better if I stupidly did it myself, rather than a dreamhost mistake or external hack, either of which would represent ongoing and hard-to-calculate risks. So I’ll tentatively accept the “shot myself in the foot” theory unless some additional info suggests otherwise. Thanks, guys.


#11

Any clues from .bash_history?


#12

if you were logged in at that time, try this: history | grep chmod
see if there’s anything in there like chmod -R 755 * or something


#13

No luck guys – I’ve been doing a lot of typing since then, chmod’ing stuff back the way it’s supposed to be, and Wednesday’s history is long gone. Should have thought to look then, or at least saved a copy, but it hadn’t occurred to me I might be to blame.

And an interesting side note that just maybe I’m not to blame… I have two public cgi’s that let people put math expressions in html using img tags, e.g., to see one of them, click http://www.forkosh.com/mathtex.cgi?\frac1{\sqrt{a^2+b^2}} New images are cached, and those two cache directories have 100,000 and 300,000 images, respectively, in them. All cached images created prior to 2012-02-15 23:00 were chmod’ed 755 at that time, and all created afterwards are 644 like they’re supposed to be. But trying to chmod them all back to 644, just gives “Argument list too long” when bash tries to expand the asterisk. So, arguably, I couldn’t have done it myself from the shell. And I’d definitely have noticed if anything I typed tried to do it. So it’s still somewhat of a mystery (at least to me).


#14

interesting. not definitive, but interesting. they could all be chmoded with the find command, or xargs, without giving an error. you’d have to really mess up to do that by accident though. i have a hard time doing it on purpose on the first go!

I think the new/old files might be a red herring though. i’m a not completely sure, but I don’t think chmod changes the umask, so while all files may have had their perms changed, that doesn’t mean new ones won’t have those some perms if the umask is set. I think that’s how it works…

Anyway, one peculiar thing is the time. The odds of mistyping and chmoding everything exactly on the hour is slim. Seems like something a cronjob would do… You don’t have a cronjob set that only runs once a year on Feb 15 @ 2300 do you? Or some long enough time that you would have forgotten the last time it ran?


#15

Yes – somebody else’s (with root privileges) runaway cron job; that’s my new best (and only) guess. Very astute of you to interpret the time that way (I’d noticed it but hadn’t made anything of it). I have no cron jobs of my own, and occasionally do use find to actually find files, but never to -exec anything. So, like you said, I’d have had to “really mess up” to accidentally do anything with the observed effect, which is why I didn’t feel your earlier chmod -R conjecture was very likely. Does dreamhost have any reason to go around chmod’ing bunches of files, like maybe under logs/, and maybe messed up somehow? Do they routinely run any kind of jobs at 11:00pm PST?


#16

You were logged in at that time, though, right? So we can’t rule that out, especially if the IP address at that time belongs to you.

I don’t think DH’s cronjobs work on the user’s dir because all the logs are actually links to files in another dir. I guess they could do something like that in an emergency if an attack left a bunch of files as 777, but I would hope that they would do that conditionally, not touching files that had not been attacked. Again, though, it’s just speculation.

Going with Occam’s razor, if you were logged in at the time of the change, I think the likelihood is that somehow it’s connected to something you or one of your apps did. Given the time, it could have been an automated process. For example, I know wordpress sets up its own cronjobs. If you have something like that running in the same user, a rouge plugin could have chmoded everything for that user.

Any plugins or the like which could have done it?


#17

Yes, I was logged in at that time, so have to agree (especially since nobody else has replied to report a similar experience) the circumstantial evidence best suggests me. I want my lawyer. And it almost certainly looks like me logged on – I have verizon dsl and, from last, whois 98.109.229.53 is verizon’s (and my modem’s currently assigned 98.109.234.92).

But, no, don’t have wordpress or any third-party plugins or any automated processes. I do have various of my own cgi’s that are documented online and intentionally publicly accessible, but they know nothing about chmod-like stuff. And umask, from your earlier reply, doesn’t seem to be any kind of issue, either, as far as I can tell. And assuming no typo within reason would have that effect, I just see no conceivable way it happened. Hence the quandary: must be me, can’t be me. I’m stumped.[hr]

Most suspicious thing I could find was a core file from one of my cgi’s (but timestamped 2012-02-16 14:28, apparently after the incident). From it I gathered REMOTE_ADDR=94.142.134.155, apparently someplace in Latvia. So I added a deny from 94.142.134. but have no idea what, if any, good that might do.


#18

It’s an interesting puzzle though. Obviously you are the type of person who knows what he’s doing, so the odds of this sort of massive mistake are slim. The odds of DH making a mistake… well, let’s not think about that too much but in general I’d say the odds are low.

I assume you’ve checked the logs for that time period? Was there any unusual activity? One possible solution may be that one of your scripts has a heretofore unknown bug which has somehow been exploited? Your diff shows there are no new files or changes, so perhaps the exploit is only minimally useful? Of course, chmoding to 777 would have been more useful if it was due to an exploit…

hmmm…


#19

I checked the logs as best I could, within reason. Those math rendering cgi’s get lots of hits. Within last seven days, including problem period, Successful requests: 4,030,448 (1,653,242), Distinct files requested: 209,669 (47,995), Distinct hosts served: 128,787 (7,825), etc. Needle in ha(ysta)ck.


#20

a little awk magic might help. it should be around the chmod time, and I would guess it’s going to be a crafted set of parameters in the url if it happened that way. You could get i list of unique request for the hour before and after:

gunzip -c ~/logs/WWW.DOMAIN.COM/http/access.log.2012-02-15 | awk '$4~/15\/Feb\/2012:2[23]/{a[$7]++}END{for (i in a) {print i}}'