Hunting the Wild Memory Creep Part 2: Understanding Memory Limits and Usage

We’re still up (2day, 7:59).
I have not solved the Memory Creep issue.
We’ve been running hot at about 90-95% since yesterday, late afternoon. I cant watch watch top any longer! :slight_smile:

I’ve been reading up on, and asking questions about, Linux Processes and I think I’ve got a bit of a better understanding. Maybe. At least as much as a few days can give.

My next set of questions (and these may be DH specific) are

[color=#FF4500]True or False, explain[/color]
On Shared Hosting, if I understood it correctly, there was a process that killed off rouge processes but left behind their orphans. These orphans grew up revenge their parents by eating all the memory and crippling the system.

[color=#FF4500]Question 1[/color]
What happens on a VPS when you hit the limit? I haven’t experienced this yet. I’ve restarted Apache (from the Panel) every time it get’s too close to the limit (1GB) in the wee morning hours. I’m too worried about letting it cause some type of crash if left on its own (like happened to us on Shared Hosting, one of the reasons we moved to VPS). I’ve been mighty close to the limit on and off for a few days, but have just let it alone. It’s been at 90-99%, and even maybe hit it (is that possible)? But I don’t know because I don’t know what happens when you hit the limit.

Can someone tell me what happens? Is there a killing of processes? What makes the site actually go down. Is it the same mechanism / results the same as on Shared Hosting?

Since there’s no Swap to disk on DreamHost for users, does this mean it always go down when it hits the limit? Does it ever recover before crashing? I’ve sworn I’ve seen it hit the limit but still stay up. Maybe not. I’m interested in knowing the actual logic behind this.

[color=#FF4500]Question 2[/color]
Why does the Memory used never drop down again once users have left the site?

Let’s say after we first start up (after an Apache reboot) we quickly run up to ~3-400,000(out of 1GB) with normal user usage. We slowly begin to creep up over the course of a day or 2 until we’re running close to the limit at ~900,000. And maybe like is happening currently, we run close to 1GB for going on 36hours. So why at 3am, when almost nobody is on the site, doesn’t the memory usage drop back down to the 3-400,000 range?

If I look at the processes running at 3AM, there are less running, a few apache2 a few php54.cgi, and the assorted other system ones. Nothing too crazy, non seeming to be hogging memory. The apache2 and php54.cgi PIDs recycle, except for the root apache one that spawned the rest (which I assume is normal that it stays in existence for so long…maybe not!?)

16890 root      20   0  148m  24m 9724 S    0  2.5   1:40.05 apache2-ps41841

So why doesn’t it drop back down? So what is holding on to the memory? Or what’s not cleaning it up? And how do I find it!? If i look at processes that have init (1) as their parent, in search of Orphans, I really only see that apache one mentioned above, and other system ones…but none seem to be using all that much memory according to the [i]top[/a] command - though they’ve been there for a long time.

6671 1 top
16787 1 dbus-daemon
16806 1 /usr/sbin/sshd
16812 1 rsyslogd
16843 1 runsvdir
16856 1 cron
16860 1 atd
16890 1 /dh/apache2/apache2-ps418417/apache2-ps418417-httpd
16980 1 /usr/sbin/inetd
17113 1 /usr/lib/postfix/master
17130 1 /usr/bin/python
25417 1 [ksecd]

The command line snippet used to get that above for those interested:

ps -elf | awk '{if ($5 == 1){print $4" "$5" "$15}}'

This is very interesting and has helped calm some of my paranoia regarding the almost always being at the Memory Limit.

linux ate my RAM

Does this apply to how DreamHost servers are configured as well?