Apache Benchmark, MySQL and Dreamhost - seeking a balance


#1

One of my clients has a vps on DH.
It hosts multiple mysql-enabled sites including Drupal and Wordpress sites.
I started getting these ‘Packing On the Pounds’ emails at one point beginning last year. (See ‘Attention Required’ email quoted at the bottom.)

I followed all the suggested advice in the links from the email for all users on the vps immediately after having received it. If you have a lot of sites, including dev sites and abandoned or old projects, you may understand that this can be pretty tedious.

  1. I couldn’t find anything out of the ordinary in the logs. If anything I noticed how truly quiet our domains are at that time of night. I’m not an expert, but I used their tools as they suggested. It might help if they posted a video on youtube showing examples of what to look for, both for vps and shared hosting.

  2. Apache Benchmark (ab): I started playing with apache benchmark because I started getting gateway timeouts. The site would simply forget where I was, especially on heavily mysql rendered administrative pages in drupal. I compared what what happening in the VPS with several other clients’ accounts that I work with on DH. These are recent ab ‘request per second’ (rps) results for various sites since I talked with DH about it. In my estimation, benchmarks doubled (or better) a few days after talking with DH about the problem:

A list of users and sites apache benchmark requests per second (rps)
User and site names munged. I’d rather not have everyone ab all the sites. I work with.

user 1 on vps
[list][]siteone.com: 40 rps (drupal 6)
[
]sitetwo.com: 9.18 rps (wordpress)[/list]
user 2 on tehran
[list][]sitethree.com 32 rps (drupal 6)
[
]sitefour.com 40 rps (drupal 7)[/list]
user 3 on masseria
[list][]sitefive.com: 32 rps (drupal 6)
[
]sitesix.com: 6 rps (wordpress)[/list]
user 4 on galactus
[list][]siteseven.com 44.66 rps (home-brew LAMP)[/list]
user 5 on alouettes
[list][
]siteeight.com 198 rps (static html)
[*]sitenine.com 353 rps (static html)[/list]

Previously, siteone.com was coming in at an average of 11 rps. Sitefive.com at about 22. A few things are apparent. First, static pages render quickest. That’s as it should be, since no demands are made on the mysql server cluster. The drupal sites are set up to cache pages for anonymous visitors, so these sites should render relatively quickly. Even with page caching, we’re getting requests per second that are pretty low. The wordpress sites are straight out of the box as-is with a few popular addons. In one case, it’s a wordpress one-click install. These typically don’t have page caching set up. I imagine most of the wordpress sites hosted by DH are not cache optimized. That means a lot of sites are running really slow on DH. If any of these wordpress sites get any real traffic at all, DH servers can’t support the load.

I’m trying to find a balance. What’s fair to expect from DH in the way of rps for a typical LAMP site using typical open-source software? Yes, it can be installed. Yes it technically works. But 9 rps for a typical wordpress site? 40 rps for a cached drupal site? Is this reasonable? Last week drupal cached sites were 11 rps. Where is the balance?

  1. Since the email is about exceeding memory requirements, I decided to take a closer look at RAM. I created a bash script to record ‘free -m’ on five minute intervals. Then I graphed the data from mysql, showing one and one-fourth day’s most recent data. (See graph below.) I noticed the largest memory use was at midnight. I had a script running file backup at that time, so I moved it to 2 am. For the next night, it shows up similarly now at 2 am, confirming that the cause of the large drop in free memory is from the backup. Why it should take over an our to backup sometimes is still eluding me, since it generally takes a few minutes if I do the same script from shell. Nevertheless, the data in the graph indicate that the cron file backup script is memory-intensive. However, I received the email several hours before midnight server time, so the backup seems less likely as a culprit. Some people may say it’s due to drupal’s managed cron script. I would concur, except that all cron tasks had been removed some how by DH several months ago. I have not re-enabled them since. For now, I run the drupal cron manually.

Something is causing spikes in memory in one of my sites - probably drupal - that makes the vps have to reboot. I really don’t want that to happen, but it is. Now, if it happens again, I’ll have this finer-grained recording of events leading up to the problem. Hopefully, it may give me a clue as to when the spike is happening on a finer grain than is available from the ‘Manage Resources’ graph on the VPS panel interface. I can check my logs and figure out what may be the cause.

One thing I’m curious about, I’ve highlighted in yellow. After the backup has finished, free RAM seems to take almost all day to recover. Is that normal?


(Incidentally, this forum won’t allow me to use the img tags to allow this to be viewed in the post. The img tag is available from the wysiwyg interface. I just can’t use it. I get an error upon preview that says ‘Your post contained prohibited content.’ Remove the img tags and it’s fine. What’s that about? )

  1. Having emailed back and forth with DH about this for the last couple weeks (either they’ve been slammed, or they’re just tired of hearing from me) they’re suggesting I add on mysql vps. Of course that’s more expensive. Since mysql is already served on a separate cluster, I’m not sure the lag in rps will be any better. For all I know, the mysql cluster is on a different campus altogether than my web server. If mysql responses were faster, would it still cause the memory leaks? Would my vps still exceed its memory limit? Right now we’re paying $30/month for web vps. I’d been considering the option of halving the RAM, in essence paying $15/month for that and adding mysql vps at $15/month, bringing the total back up to $30/month. In robbing Peter to pay Paul, am I just borrowing trouble?

  2. At $30/month that we’re already paying, I’m guessing there may be VPS hosting situations out there that are more drupal-friendly, less likely to reboot the vps and more likely to have a much higher apache benchmark.

I’d be grateful for any feedback.

[quote=‘Attention Required: Your DreamHost VPS needs help!’] Nobody likes to hear that they’ve packed on a few pounds, but we wanted you to hear it from a friend.

Your DreamHost VPS, ps#####, has just exceeded the memory allocation that you’ve established. If left unchecked that behavior could begin to negatively impact the VPS services of every other customer on your server.

As a result we’ve had to reboot your VPS, effectively restarting your virtual machine. You may experience several minutes of downtime while your services come back online.

Based on the usage pattern that we’ve seen today, it’s clear that you’d be best served by either working to reduce your memory footprint or simply increasing the amount of memory available to your VPS.

We’ve put together some documentation that can help you troubleshoot and even reduce your current memory usage:
http://wiki.dreamhost.com/DreamHost_PS_Troubleshooting
http://wiki.dreamhost.com/PS_Optimization

Remember, you can visit https://panel.dreamhost.com/index.cgi?tree=vserver.usage to scale your memory allocation up (or down) whenever you’d like!

Strictly speaking, you don’t HAVE to take any action right now, but if your memory usage continues to routinely exceed the allocation level you’ve specified, you may experience more brief periods of downtime as your services are restarted. You may also get sick of seeing this message in your inbox. Unfortunately there’s not much that we can do about either, so we hope you’ll consider upgrading!

Thank you for choosing DreamHost![/quote]