Periodic slowdown fixed by Apache reset...but it keeps hapening


#1

This is in response to the following thread, but more general then just perl scripts.

I am new to Dreamhost, and after setting up several domains on my account, I would notice unbearably slow response times for any script to run, cgi, PHP or otherwise. This was mainly affecting Joomla and Wordpress sites. Even some freshly installed sites with nothing more then a few add-ons and a custom theme. I would contact support, they would see some processes hung in Apache, reset the instance, then everything would be great for a few days until I began to experience the issues again.

Most recently support was pointing to my sites as the culprit, but none of them are actually getting any real world exposure or traffic and I am at a loss as to how a site that has hummed along without any issues on another hosting provider has now all of a sudden begun to have performance issues.

I am very curious if anyone else has experienced this issue?


#2

It could be bad plugins/themes/add-ons. They can get annoying really fast.

For WordPress, there’s a (heh) plugin that can help you find the bad plugins: http://wordpress.org/extend/plugins/p3-profiler/


#3

[quote=“Ipstenu-DH, post:2, topic:58840”]
It could be bad plugins/themes/add-ons. They can get annoying really fast.[/quote]

It’s not a plugin issue. A simple <?php echo 'hello'; ?> script runs slow. It has nothing to do with Wordpress.


#4

Maybe, maybe not. If one person in a room is really loud, the whole room is loud :slight_smile: If you have something on WP (or any other app on your domain) that’s running hot, it will impact everyone.

There’s also the possibility you have a bad neighbor (ie. someone else on your server is being noisy). You can ask to be moved to another host, if you want, but I always like to make sure I’m not the bad guy, first. (I was, more than once.)


#5

I see your point. :slight_smile:

Well… in my case I have an issue with Perl scripts. At the same time static HTML pages are fine.

At the same time because DH support response is extremely slow (more than 24 hrs) then it’s not possible to investigate anything. How it is possible to investigate an issue when it’s already resolved (by itself???)? We just waiting for the next incident, logging another call, and again DH support comes back when another issue already disappears. Sure they reports “no issues found” and close the ticket. But at the same time the issue is still there. It’s really frustrating.


#6

You can check yourself if it’s a load/memory/cpu/etc issue if you set a user to have shell access.

Just SSH in and issue the top command.


#7

I hear your frustration :frowning: We’re doing our best to get to everyone’s tickets as fast as we can, but we are taking longer than we’d like right now.

sXi has great advice with top, but if you need help, and have livechat available on your account, maybe that would do to help in the ‘now’ of the problem?


#8

I continue to see this issue with various domains on my account. How can I tell which domains are on which server? I am also running the TOP command in a terminal for two of the accounts that have domains with issues and I don’t see any process that seem to be running rampant, yet the sites are slow. In fact, I see only 3 process running (sshd, bash & top), occasionally I see PHP run but only briefly. I am trying to make sure it isn’t me, but I honestly don’t see how this site has anything of any significance running on it to amount to spit when it comes to consuming server resources: http://icareseniors.dreamhosters.com
[hr]
sXi, I just have to thank you for the top command suggestion, very cool. I can now see the process that start and end in the blink of an eye for a site that load very quickly, and the lack of any processes for the site that loads like a toad.


#9

I sat on top (heh) for a little while on your server and it runs great. Loads of around 1 are wonderful for that server. I also checked your server locations, and the DB servers and fileservers are in the same datacenter.

If you go to https://panel.dreamhost.com/index.cgi?tree=billing.accounts& you can see what servers everyone is on.


#10

I finally came to the forums looking for answers after my sites started behaving poorly again. The first time, the problem disappeared by the time Dreamhost Support responded 46 hours after my report.

It seems primarily perl related. I’m running on Wythe and the load average seems to vary between 14 and 30 with no obvious differences.

Static pages load fine.

The problem is apparent even from Wythe running Lynx or wget.

Here is my slowperl.cgi script:

#!/usr/bin/perl

use CGI qw/:standard/;

print header;
print start_html('slowperl.cgi');
print h1("Hello, World!");
print "<tt>";
#print `cat slowperl.cgi`;
print "</tt>";
print end_html;

I’ve noticed something curious. If the cat... line is commented out, the delay is exactly 30 seconds. If the cat.. line is called, the delay is 60 seconds.

Here’s the output of wget http://borowski.net/slowperl.cgi WITH the cat:

--2013-03-27 15:01:37--  http://borowski.net/slowperl.cgi
Resolving borowski.net... 208.113.213.52
Connecting to borowski.net|208.113.213.52|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 560 [text/html]
Saving to: "slowperl.cgi"

     0K                                                       100% 23.0M=0s

2013-03-27 15:02:37 (23.0 MB/s) - "slowperl.cgi" saved [560/560]

and without:

--2013-03-27 15:01:37--  http://borowski.net/slowperl.cgi
Resolving borowski.net... 208.113.213.52
Connecting to borowski.net|208.113.213.52|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 560 [text/html]
Saving to: "slowperl.cgi"

     0K                                                       100% 23.0M=0s

2013-03-27 15:02:37 (23.0 MB/s) - "slowperl.cgi" saved [560/560]

The 30-second thing seems to be very repeatable so far.

From the command line, slowperl.cgi runs in a jiffy:

>time ./slowperl.cgi
Content-Type: text/html; charset=ISO-8859-1

<!DOCTYPE html
        PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
         "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en-US" xml:lang="en-US">
<head>
<title>slowperl.cgi</title>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" />
</head>
<body>
<h1>Hello, World!</h1><tt></tt>
</body>
</html>0.080u 0.000s 0:00.20 40.0%      0+0k 0+0io 0pf+0w

If I replace the cat... with a set of three print date; lines, the delay is a nice even 2 minutes; 30 seconds for the perl script itself and 30 seconds for each backtick shell command. And the output is (as I expected based on the incremental delay):

Wed Mar 27 12:22:24 PDT 2013
Wed Mar 27 12:22:54 PDT 2013
Wed Mar 27 12:23:24 PDT 2013

What the heck?!?

I’m going on 21 hours for case #5676684 now. I wish I could append the information about the 30 second delay increments…

And thoughts from fellow customers or DH staff would be greatly appreciated!

Brett


#11

Hi, I am also experiencing something weird with my cgi scripts:

This is my cgi script:

[php]
#!/bin/sh

echo "Content-type: text/html"
echo
echo “Hello, world!”
[/php]

[php]
[william-floyd]$ time curl "http://www.deantoni.com.br/info.cgi"
Hello, world!

real 0m30.045s
user 0m0.008s
sys 0m0.012s
[/php]

Why is it delaying 30 seconds?

Thanks!!


#12

Request to be moved from that server.


#13

It looks like I’ll have to do that. How do we ensure that we don’t end up on another server with the same problem though?

I’m on wythe. What servers are all the other folks having the same problem on?

And is switching servers just not a big deal and completely seamless? (My switch from CA to VA seemed to go OK.)


#14

My transfer request went okay.

They don’t randomly dump you anymore if you request to be moved.


#15

I’ve done some initial investigation and have forwarded the issue to our system administrators. Something very strange is going on here.

For the time being, it appears that there’s a simpler fix than moving you to another server — even just restarting the Apache service may be sufficient to solve the problem, at least temporarily. I’ve done this for deantoni’s Apache service, and his script seems to be behaving normally now.


#16

Hi Andrew,

It fixed the delay for some time, but apparently the problem is back =/

What do you suggest?


#17

This is a known problem. I’ve experienced it and there are several threads on this forum documenting it. Almost everyone who has documented it has created minimal scripts such as the ones posted here, i.e. just echoing ‘hello world’, and still see lags of 30 or 60 seconds.

Unfortunately, DreamHost doesn’t seem to have a good answer to why this is happening. Requesting to have your apache instance restarted or moved to a new instance will help, but it doesn’t address the root problem.

I would love to see DreamHost make some real progress on this. I’m not experiencing it at the moment, but the spectre of it returning is always on my mind.


#18

Crud. I was afraid of that.

But I’ve got a solid theory as to what’s going on. I’m going to talk to some people tomorrow about what we should do about it.


#19

I just started seeing something similar to this, it started during the maintenance period on Wednesday, and has been continuing. I have made no configuration changes at all to my Wordpress site, but now when I try to load a page, it either takes about 1-2 minutes to load, or it loads part of the html, pauses for about 1-2 minutes, then finishes or times out, depending on the browser. When I try to load from a mobile device it never even loads, either I get “the server has stopped communicating” or I get a “504 - Gateway not responding” error.

When I run “top” I notice that there are always about 4 php processes running at all times, but the server load is in the 12.0 to 25.0 range, which is far too high. Usually loads like that make even shells unresponsive, but I have no problem getting in via ssh and running commands via shell. Also, the wp-admin pages seem to load fairly quickly, which leads me to think it might be something my Wordpress site is relying on that the admin doesn’t need.

I have two trouble tickets in on this, but it sounds like it may take a couple of days to a week before they get to looking at it. If it takes a long time to get a trouble ticket answered, how can I ask for an apache restart?

BTW, here’s what a “top” output looks like ( the php processes hang around ):

top - 09:00:51 up 152 days, 6:13, 2 users, load average: 21.48, 14.11, 12.54
Tasks: 7 total, 1 running, 6 sleeping, 0 stopped, 0 zombie
Cpu(s): 67.6%us, 10.4%sy, 11.5%ni, 3.8%id, 4.8%wa, 0.1%hi, 1.8%si, 0.0%st
Mem: 24691968k total, 23910848k used, 781120k free, 791860k buffers
Swap: 8000328k total, 1961568k used, 6038760k free, 5618240k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1241 cptnerd 20 0 18888 1132 900 R 0 0.0 0:26.62 top
48922 cptnerd 20 0 242m 50m 8580 S 0 0.2 0:03.89 php53.cgi
54177 cptnerd 20 0 241m 49m 8532 S 0 0.2 0:00.87 php53.cgi
54524 cptnerd 20 0 239m 47m 8524 S 0 0.2 0:00.53 php53.cgi
54563 cptnerd 20 0 240m 47m 8520 S 0 0.2 0:00.56 php53.cgi
64357 cptnerd 20 0 64036 1676 1048 S 0 0.0 0:01.06 sshd
64363 cptnerd 20 0 94280 1440 592 S 0 0.0 0:00.01 ksh


#20

that depends on your server, most DH shared seem to be 16 core processors, given that those loads aren’t real high.

Open a ticket, tell them you troubleshooting a page load time problem and ask them to restart the apache instance. Be sure to give the domain or sub-domain name, if you have more than one on your account because they may or may not (probably not) be on the same apache instance