Thanks for the reply. I am hosting around 60 sites on VPS, although all of them but 2 are tiny and don’t get much traffic.
Unfortunately, I have very little experience in profiling applications or doing any backend apache stuff. I don’t know how to manage any of the server-end stuff. I had to teach myself how to use sudo just to fiddle with the apache conf file to fix the server crash issues. That “New Relic” software looks like it costs $150/mo, which is too much for me. I’ve been trying to find a good open source product, some of these might also require “sudo” access which won’t be available on VPS soon… Although, it would be nice to learn how to find some of this server performance information and learn more server-end stuff.
Currently, I use the “top -c” shell command to view the running processes for each site and memory usage. When I write a script, I’ll run it and watch it here. I think the problem Dreamhost has is it sets a high limit on the number of processes that run (the php54.cgi processes). So I had to set a hard limit of 6 processes.
What happens: visitors view the site, processes start up and they don’t last long and usually don’t use too much memory, then they sit around in “sleep” mode until the expiration time is up. During normal traffic levels, I assume Dreamhost is trying to make the site load as FAST AS POSSIBLE for all visitors, so it spawns 8, 10, 12+ processes, with many of them ending up in “sleep” mode. Each php54.cgi process is usually around 20 MB, but it can go up to 80MB each when processing scripts for loading up pages where images need to be resized for thumbnails. Too many processes are spawned, even while some processes are still “sleeping”, then the memory gets maxed out and server crashes…
I found that limiting the processes to 6 (80MB * 6) prevents a memory crash and these 6 processes allows the site to load up with reasonable speed. The drawback is it slows the site down (slightly) during heavier traffic loads when all 6 processes are running hard trying to handle all traffic… I found that 6 processes running frequently with reasonable loading time, is better than many processes running “not so often” and using up lots of memory (basically a site that loads up super quick for everyone, but crashes every 30 minutes).