Dos or enormous delay loading web pages on strathclyde?


#1

Anybody else seeing this problem?

I can ssh into my shell account and do stuff there without any problem or delay.
But pointing browser to my homepage takes pretty much forever (several mins) to load.
Tried order deny,allow; deny from all; allow from “me”; in .htaccess, and that seems
to clear things up (for “me”) after ~15 minutes. But remains slow for the first 15 mins.
So I’m guessing that suggests dos.

Tried panel.dreamhost.com to view logs and figure out who might be doing what.
But I can’t see any way to determine immediately recent activity.
I’d like to see exactly who’s accessing the site in pretty much realtime.
Is there any way to do that? Any other suggestions diagnosing dos (or diagnosing
whatever else it is that might be going on)? Thanks.
[hr]

[quote="forkosh, "]
Anybody else seeing this problem? <>[/quote]
Problem seems to have disappeared (now that I posted it),
and .htaccess restored to original. But it >>definitely<< occurred!!!
That was between about 11pm 10/29/14 thru 1am 10/30/14, Eastern daylight time
(localtime in New York City). Anybody else notice such a thing around that time?


#2

Raw logs for your domain are available via ssh/SFTP at /home/USER/logs/DOMAINNAME/http/ (hint: you can NOT navigate there via FTP, you MUST use a secure protocol or the path won’t exist or will be empty.)

As far as a problem with the server itself, from what you said probably not. Of course one of the things worth checking immediately is the servers load averages using uptime or top. Likely you won’t find a problem there, dreamhost must pay attention because you don’t see many complaints in this forum about overloaded shared servers. If you do find high load averages, you likely can’t do anything to fix it other than open a ticket.

So let’s move on the the next layer, the webserver (or apache instance). This is likely where the issue is/was. Apache instances are also shared. In other words, each shared server has multiple shared apache instances. This is also the level at which IPv4 IP’s are assigned, that is each website that shares an apache instance also shares an IPv4 address. A DDoS attack against another site could be felt here, but that shouldn’t last too long tho because dreamhost defends against them routinely and has protocols in place to help protect other customers.

There was a long running “apache glitch”, which I hope using the word “was” is correct–we certainly hope that didn’t migrate from debian to ubuntu–no word from dreamhost about the status of that bug. Again tho, if the problem is at this level you probably can’t fix it, except to open a ticket. You can only see the http logs for your domains, if someone elses site on the same apache instance is under attack you can only guess.

You can use tools, like this one, to find guesses as to what sites are sharing the same IP, beware tho the way data is collected tho makes it incomplete and/or out-of-date by definition–manually verify the IP of any site you might be sharing with, it may have moved or no longer exist. That said, I did use information gathered via this method once in a support request to move my site to a different apache instance. I suspected by name that one site might be susceptible to attacks on its subject matter. Changing the apache instance in that case did solve the problem, in retrospect tho it might have just been that apache glitch we already talked about. Along the way I learned that if you request that your site be moved to another apache instance to eliminate that from a list of possibilities that support is very cooperative and even will hand pick the new apache instance.

Did you open a ticket? Of course if the problems not happening anymore support can’t very well chase it. If it’s an intermittent bug you have to try to find the pattern.

If you have cloudflare turned on, that changes everything–most users requests don’t reach the server directly.

[EDIT: my memory failed… I don’t think this was you, I can’t find the thread I was think of tho… I’ll leave it since it’s good info for someone reading along anyway.]
Also, since you mentioned elsewhere that your wife wants you to set up a blog for her, certain combinations of wordpress plugins and/or theme’s can cause issues like you described in your post. Not everything plays nice together. If you need plugin X for reason Y, that may preclude you from also being able to use plugin Z. If your dealing with diagnosing a wordpress domain, then first thing to do is disable all plugins and change the theme back to twenty-fourteen. If the problem goes away turn things back on one at a time until you find where conflicts start, and/or just ask around. Most of the time if you ask in this forum or over in the wordpress forum for specific plugin compatibility issues you will get alot of help.


#3

Thanks for the detailed suggestions, LR. The raw logs aren’t easy to interpret. I’m going to write a short script to sort them by that leading ip, then do a uniq -c, and then re-sort by its --count. Applying that to a head -10000 (or some number) of the current log should show what’s going on at any particular moment.

I should have mentioned that I had checked top at the time. None of the 8 cpu’s on strathclyde were at 100%, but they all ranged from ~30% to ~70%, so pretty busy but nowhere near explaining the delay. By the way, the problem did re-occur from ~2am-~3am EDT on 10/30/14. Haven’t seen the problem again (but haven’t checked all that often). By the way#2, although the “deny from all except me” did seem to work when I first tried it (and mentioned it in preceding post), it stopped working after a while, and the problem resumed. Maybe the problem just coincidentally subsided for a while at that time; I don’t know. But that sounds to me like another possibility you suggested: a DDoS against another site on the same server. Your http://www.yougetsignal.com/tools/web-sites-on-web-server/ tool showed 18 sites including me. But, again, I don’t know anything for sure except the symptoms I observed.

Regarding apache glitch: I’d never observed anything like this before, so have apparently not been affected by it previously. So I’m guessing it’s not the cause, although maybe somebody just started to exploit it. Again, totally beyond my knowledge.

Didn’t open a ticket – wanted to first see if anybody else was seeing the same thing. Apparently not, and, like you pointed out, since I’m no longer seeing it either, probably a waste of everybody’s time to report it further.

Regarding cloudflare (no, not turned on by me; don’t even know what it is:) and wife wanting blog, that’s gotta be other people (is that what you meant by “memory failed”?).
Thanks again,


#4

Just in case anybody’s interested in that “script” mentioned at the beginning of my preceding post, it’s just the following one-liner
awk ‘{split($0,ip,"- -"); print ip[1]}’ | sort | uniq -c | sort -nr
I saved that in a file logs.sh in my personal /home/username/bin/ directory that’s on my $PATH,
then cd’ed to the logs/domain/http directory pointed out by LR previously, and then just
head -1000 access.log | logs.sh | less
to see something like
445 213.37.5.107
68 81.252.118.57
54 89.7.47.152
41 31.192.231.21
31 89.173.28.115
25 114.113.197.132
etc., showing you that 213.37.5.107 has accessed your site 445 times out of the most recent 1000 accesses (or change that -1000 to any other number). Hopefully, that kind of thing will be useful. Any suggestions how it might be improved?

P.S. How long’s it gonna take to get rid of that “windlee” spammer? (not that I have anything against Las Vegas escorts:)