How do I find out what is eating resources?

vps

#1

Hello everyone…

I’ve been a DH customer for quite some time and when the chance came to switch to VPS I did so without blinking.

Now, I’m not a full time development person. I have some sites that I run as favors and I have one for my business that helps us bring in customers, but I don’t sit at the computer all day.

I am a frequent receiver of the “exceeded memory” email from Dreamhost even though I have easily tripled the memory allocation without adding any real traffic to the sites I run.

What is the best way for me to figure out what in the heck is eating all my memory? Where do I see a comparison of traffic? How do I know where the bandwidth is going?

I’m seriously considering switching to Cloud Server at RackSpace just so I can build in some more fine-grain controls of my servers. My total bandwidth usage for a month is only 2.5Gb but somehow I’m eating close to 1Gb memory constantly.

Any help is appreciated.


#2

The most helpful tools from http://wiki.dreamhost.com/Finding_Causes_of_Heavy_Usage:

"If you have multiple domains on and on a PS (PS only!) run this command to get all traffic for all domains on the PS:

and then from there, cd to the heaviest hit site log dir by using

"and this command is even more useful in some cases as it specifically targets the last 10,000 hits:

and
This command is great if you want to see what is being called the most (that can often show you that a specific script is being abused if it’s being called way more times than anything else in the site):

That wiki also includes info about blocking abusive IPs if necessary! If anything seems out of the ordinary or abusive still, you can write to our support team via your panel and request an account scan from our Security team.


#3

Thank you for your help, although I’m not sure I know what I’m looking at…

[psXXXXX]$ tail -10000 access.log| awk '{print $1}' | sort | uniq -c |sort -n 1 123.125.71.60 1 208.80.194.173 1 208.93.105.250 2 173.199.116.75 2 208.167.230.51 4 211.212.39.237 6 173.199.154.160 33 166.205.55.40 1334 69.163.225.246

Can you explain to me what the numbers in the left column mean?


#4

What this report is is a list of ip addresses and what they are doing on your site. What this say is ip address 69.163.225.246 had 1334 log entries. If you look ip this ip address (google: what’s my ip address and find a website that lets you research the location and name of an ip address) you’ll find out it’s actually your fps at dream host. You had 8 other visitors with their ip addresses listed. If you worked on the site your probably the user with 33 accesses.

Bill


#5

the quickest way to see what your memory hogs are is to run a shell command that lists memory by process. top gives a summary that updates in real time. ps (use the command ps axuf to list all processes including subprocess… it may be more than a screen and you’ll need a pager to look through it, like less or most) …

then RSS is resident memory and if you do the list with ps you can see which process (or subprocess) is the memory hog.

by default there’s a few things enabled on dreamhost servers you may not even need or use, and you can save memory by deactivating them. mail goes to dh’s mail server by default so turn off courier (and postfix if you don’t send mail from the machine). proftpd if you don’t use an ftp server (sshd provides sftp which is better to use unless you need a public ftp server). maybe a few other things. i nixed supervisord because it does nothing i need. and jabber. and maybe some other stuff that was running by default. all that eats memory.

the scripts to start these things are in the directories /etc/rc.d where whatever is your machine’s current runlevel (which the runlevel command will tell you). you change the links for the things you want to deactivate to K01whatever from S03whatever (just do for example mv S02jabber K01jabber, it’s a lot of typing…)

i think mysql is enabled on servers by default. but if you use dreamhost control panel to set up your database, you may not even be using your local machine’s mysql at all, so that can go too. (though it’s (much) faster if you do use it, there’s an option on the panel to move your databases there). if you don’t definitely disable that too.

so after you disable a bunch of stuff, reboot your machine, run ps axuf again to see what is still running, you should be seeing less memory gobbled up.

about getting httpd to use less memory, there’s tons of web pages on that. you want to run less servers basically. i forget how to configure it off the top of my head, just google for that, there’s guides out there.