Hunting the Wild Memory Creep Part 1: Understanding Apache Processes


#1

Greetings good DreamHost users out there!

The Setup
I have inherited a custom PHP site (in quite a non-organized state) that has recently been crashing due to hitting the memory limit (503 errors). Error logs at the time showed that we were “hitting the max fcgi spawn process limit for the server.” (DH Support). There was no code change in Production, and no sudden increase in traffic, so not sure why after many years of being hosted here, this began surfacing. We moved off Shared Hosting to a VPS (this is last week). Moved our DB server to the same Data Center as Web server. I followed through with several of the optimization steps in the DH Wiki. We are up, but the memory keeps slowly climbing up until I have to bounce Apache. It takes from 1-2 days to hit the max. So, the question becomes WHAT is causing this (and maybe also WHY NOW?), but more importantly, WHAT so we can fix it.

The Possible Suspects
[list]
[]PHP scripts
[
]MYSQL queries
[*]Apache config
[/list]
The site was built over the years by a developer who made it all work, but didn’t always use the best PHP practices, or just ones that are out of date. So I know we’re dealing with older, legacy code that is unorganized, and that there’s likely issues in there for sure! The same goes for the queries in there. Long, long queries, with 4 Left Joins, and complicated math inside the SQL. As for Apache, I’m guessing the standard DH setup is not the issue, but I include it here because I’m just not sure. There are other performance issues that need to be address (how assets like js,css,images are loaded, etc), but I think if we’re talking memory being eaten up, those may slow the site down, but are not the main culprits - correct me if I’m wrong!

The Hunt Part 1: Processes
So everyone (google searches, etc) says to check out the Processes running to get insight into what is eating up the memory. Often they suggest using “top” or “ps” type command line tools. I’ve been watching “top” for days now, watching the Mem: go up, and up. But I don’t know exactly how this will help me. I can see Processes come and go, every once in a while a Zombie appears and then is gone. So I guess I’m just not sure how this will help me narrow down where my issues are coming from in the code?

Looking at top:

  1. Time: if I’m looking for a Proccess that hung up, how long would it be there for? Most don’t seem to go beyond a few minutes (expect for root). 0:02:41 for instance.
  2. Mem: I don’t see any go above about 2.
  3. Command: I assume this is the name of the Process itself, or what calls it? The 2 that I see the most are apache2-psxxxxx and php54.cgi. Okay, well common sense tells me that these are Apache and PHP processes, right? Though when i run pstree I see that the php54.cgi are children of the apache2 ones.

I think I’m just confused about Apache / PHP processes and what I’m seeing here.
AND how this can help me track down areas (scripts) in my site that might be causing memory issues.

[code]top - 08:59:40 up 11:09, 2 users, load average: 0.00, 0.02, 0.00
Tasks: 40 total, 1 running, 39 sleeping, 0 stopped, 0 zombie
Cpu(s): 6.9%us, 0.8%sy, 0.2%ni, 92.0%id, 0.0%wa, 0.0%hi, 0.2%si, 0.0%st
Mem: 1024000k total, 519828k used, 504172k free, 0k buffers
Swap: 0k total, 0k used, 0k free, 408752k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16890 root 20 0 147m 22m 8248 S 0 2.3 0:16.33 apache2-psxxxxx
1403 dhapache 20 0 368m 22m 4616 S 2 2.2 0:00.07 apache2-psxxxxx
1128 dhapache 20 0 440m 21m 4772 S 1 2.1 0:00.12 apache2-psxxxxx
1127 dhapache 20 0 440m 21m 4800 S 1 2.1 0:00.08 apache2-psxxxxx
1396 dhapache 20 0 440m 20m 4460 S 0 2.0 0:00.03 apache2-psxxxxx
18974 user3 20 0 278m 20m 11m S 0 2.0 0:02.30 php54.cgi
12109 user3 20 0 277m 20m 12m S 0 2.0 0:02.20 php54.cgi
1370 dhapache 20 0 292m 19m 3996 S 0 2.0 0:00.03 apache2-psxxxxx
18975 user3 20 0 275m 19m 11m S 1 1.9 0:00.77 php54.cgi
1453 dhapache 20 0 291m 18m 3372 S 0 1.9 0:00.00 apache2-psxxxxx
1404 dhapache 20 0 147m 18m 3228 S 0 1.9 0:00.00 apache2-psxxxxx
1462 dhapache 20 0 147m 18m 2944 S 0 1.8 0:00.00 apache2-psxxxxx
1463 dhapache 20 0 147m 18m 2944 S 0 1.8 0:00.00 apache2-psxxxxx
32602 user3 20 0 274m 16m 9488 S 0 1.6 0:00.04 php54.cgi
25115 user4 20 0 272m 15m 9044 S 0 1.6 0:00.03 php54.cgi
815 dhapache 20 0 146m 15m 696 S 0 1.5 0:00.54 apache2-psxxxxx
25417 root 20 0 83108 15m 3176 S 0 1.5 0:01.81 [ksecd]
763 dhapache 20 0 145m 15m 332 S 0 1.5 0:00.00 apache2-psxxxxx
17130 root 20 0 59428 10m 1028 S 0 1.0 0:06.57 supervisord
6798 user3 20 0 130m 5712 1756 S 0 0.6 0:00.23 bash
31184 user2 20 0 129m 5552 1692 S 0 0.5 0:00.15 bash
31134 root 20 0 71360 3404 2620 S 0 0.3 0:00.01 sshd
6598 root 20 0 71360 3400 2620 S 0 0.3 0:00.00 sshd [/code]

init─┬─[ksecd] ├─apache2-psxxxxx─┬─4*[apache2-psxxxxx] │ ├─apache2-psxxxxx───4*[php54.cgi] │ ├─8*[apache2-psxxxxx───4*[{apache2-psxxxxx}]] │ └─apache2-psxxxxx───3*[{apache2-psxxxxx}] ├─atd ├─cron ├─dbus-daemon ├─inetd ├─master─┬─cleanup │ ├─pickup │ ├─qmgr │ ├─smtp │ └─trivial-rewrite ├─rsyslogd───3*[{rsyslogd}] ├─runsvdir ├─sshd─┬─sshd───sshd───bash───pstree │ └─sshd───sshd───bash └─supervisord

Anyhow, thanks to anyone who’s taken the time to read through all of this. I know these memory issues plague lots of folks on no matter where they are hosted, and sometimes the answer(s) is(are) not easy to pin-point. But I figure there may be some folks more experienced with the Sys Admin side of the PHP world that can help get me going in the right direction! (Part 2. coming when I figure out what it is :slight_smile: )


#2

Hi azabraxas,

We would be happy to look into this on our end if you can provide an account number or domain name. You can pm me if you would like to keep that info private.

Thanks!


#3

Thanks!

We’ve had several Support folks look at it over the last week, so I don’t want to just retread the same ground…(i.e. you may want to look at the support ticket trail).

Also, if you find anything that’s not covered in the wiki’s or in those emails, feel free to share here in the forum in case it helps others…

And even if you can’t find anything specific, shedding some light on the OP questions on Processes would be awesome!

I bounced Apache at 1AM this morning.
Limit is 1GB.
We’re hovering at around 850,000 as of writing this.

Account:
(#357833)