Too much CPU


#1

How do we know

What CPU usage I am currently using?
What CPU usage each domain use?
Whether it’s mysql or regular sites that’s causing the problem (moving my databases to hostgator will fix that)

Thanks,


#2

For really current usage information, use “top” in the shell. Otherwise:

http://wiki.dreamhost.com/CPU_Resources_FAQ

CPU usage is tracked by user, not domain, so check the user usage for the user under which the domain is running.

That will take some review of your coding, maybe some benchmark testing, etc. -

–rlparker


#3

I run top and this is what I see:

9333 teguh123 20 0 20312 8644 4812 R 1.3 0.2 0:00.04 php5.cgi
9424 teguh123 20 0 20316 8456 4664 R 1.3 0.2 0:00.04 php5.cgi
9283 teguh123 20 0 20312 8640 4824 S 1.0 0.2 0:00.03 php5.cgi
8810 teguh123 20 0 20416 9052 4964 R 0.7 0.2 0:00.06 php5.cgi
9591 teguh123 20 0 18876 6248 3956 R 0.7 0.2 0:00.02 php5.cgi
9595 teguh123 20 0 19004 6628 4124 R 0.7 0.2 0:00.02 php5.cgi
8345 teguh123 20 0 22624 10m 4908 S 0.3 0.3 0:00.05 php5.cgi
9471 teguh123 20 0 14408 4220 3232 R 0.3 0.1 0:00.01 php.cgi
9513 teguh123 20 0 16384 3116 2540 R 0.3 0.1 0:00.01 php5.cgi
9594 teguh123 20 0 15988 2248 1792 R 0.3 0.1 0:00.01 php5.cgi
7767 teguh123 20 0 0 0 0 Z 0.0 0.0 0:00.04 php5.cgi
7955 teguh123 20 0 0 0 0 Z 0.0 0.0 0:00.06 php5.cgi
8469 teguh123 20 0 14504 1428 1056 R 0.0 0.0 0:00.00 sshd
8473 teguh123 20 0 4604 1500 1128 S 0.0 0.0 0:00.00 bash
8480 teguh123 20 0 20312 8776 4920 S 0.0 0.2 0:00.04 php5.cgi
8554 teguh123 20 0 20312 8772 4908 S 0.0 0.2 0:00.04 php5.cgi
8619 teguh123 20 0 20316 8788 4908 S 0.0 0.2 0:00.04 php5.cgi
8620 teguh123 20 0 20828 9212 4912 R 0.0 0.2 0:00.04 php5.cgi
8631 teguh123 20 0 20312 8888 4912 S 0.0 0.2 0:00.04 php5.cgi
8635 teguh123 20 0 20828 9272 4908 S 0.0 0.2 0:00.04 php5.cgi
8667 teguh123 20 0 20316 8744 4876 S 0.0 0.2 0:00.04 php5.cgi
8670 teguh123 20 0 20312 8736 4868 S 0.0 0.2 0:00.04 php5.cgi
8671 teguh123 20 0 20312 8792 4912 S 0.0 0.2 0:00.04 php5.cgi
8690 teguh123 20 0 20316 8780 4908 S 0.0 0.2 0:00.04 php5.cgi
8691 teguh123 20 0 20312 8860 4916 S 0.0 0.2 0:00.04 php5.cgi
8695 teguh123 20 0 20312 8924 4916 S 0.0 0.2 0:00.04 php5.cgi
8703 teguh123 20 0 20312 8756 4896 S 0.0 0.2 0:00.04 php5.cgi
8710 teguh123 20 0 20312 8756 4896 S 0.0 0.2 0:00.04 php5.cgi
8727 teguh123 20 0 20316 8796 4908 S 0.0 0.2 0:00.04 php5.cgi
8732 teguh123 20 0 20316 8780 4908 S 0.0 0.2 0:00.04 php5.cgi
8733 teguh123 20 0 20316 8760 4896 S 0.0 0.2 0:00.04 php5.cgi
8738 teguh123 20 0 20312 8756 4896 S 0.0 0.2 0:00.04 php5.cgi
8743 teguh123 20 0 20316 8764 4904 S 0.0 0.2 0:00.04 php5.cgi
8753 teguh123 20 0 20312 8732 4876 S 0.0 0.2 0:00.04 php5.cgi
8758 teguh123 20 0 20312 8716 4868 S 0.0 0.2 0:00.04 php5.cgi
8759 teguh123 20 0 20312 8788 4920 S 0.0 0.2 0:00.05 php5.cgi
8761 teguh123 20 0 20316 8760 4896 S 0.0 0.2 0:00.04 php5.cgi
8762 teguh123 20 0 0 0 0 Z 0.0 0.0 0:00.08 php5.cgi
8765 teguh123 20 0 20312 8764 4896 S 0.0 0.2 0:00.04 php5.cgi
8768 teguh123 20 0 20312 8792 4912 S 0.0 0.2 0:00.04 php5.cgi
8773 teguh123 20 0 20312 8728 4868 S 0.0 0.2 0:00.04 php5.cgi

What does it mean?

How much CPU usage I am using total?

Why I am the only user name that shows up?


#4

You can learn more about top, including how to customize it’s output and how to interpret what you see, by running “man top” from a shell prompt (this will display the top man(ual) pages).

Add up the “%CPU” column valules to see the total % of CPU your user is using.

Because that is what TOP, run as your user on DreamHost, shows by default.

–rlparker


#5

Yes… But how do I check CPU usage?

What command in SSH would allow me to do so?

I downloaded some logs and I see the following:

Process CPU seconds user machine count average
php5.cgi 24828.5700 87.525% 103.452% 411053 0.060
php.cgi 3537.3700 12.470% 14.739% 112386 0.031
awk 0.7400 0.003% 0.003% 204 0.004
sort 0.3200 0.001% 0.001% 408 0.001
tail 0.2100 0.001% 0.001% 202 0.001
uniq 0.1100 0.000% 0.000% 204 0.001
suexec 0.0400 0.000% 0.000% 10 0.004
man 0.0400 0.000% 0.000% 2 0.020
cat 0.0300 0.000% 0.000% 7 0.004
ls 0.0300 0.000% 0.000% 13 0.002
troff 0.0200 0.000% 0.000% 1 0.020
bash 0.0100 0.000% 0.000% 1 0.010
nroff 0.0100 0.000% 0.000% 1 0.010
groff 0.0100 0.000% 0.000% 1 0.010
grotty 0.0100 0.000% 0.000% 1 0.010
sh 0.0100 0.000% 0.000% 3 0.003
find 0.0000 0.000% 0.000% 1 0.000
zsoelim 0.0000 0.000% 0.000% 1 0.000
sshd 0.0000 0.000% 0.000% 1 0.000
nroff 0.0000 0.000% 0.000% 1 0.000
tbl 0.0000 0.000% 0.000% 1 0.000
bash 0.0000 0.000% 0.000% 1 0.000
pager 0.0000 0.000% 0.000% 1 0.000
gzip 0.0000 0.000% 0.000% 2 0.000
locale 0.0000 0.000% 0.000% 1 0.000

Total: 28367.5300 100.000% 118.198% 524507
Average per day: 28367.5300 1 days
CPU percentage assumes 24000 cpu seconds per day total.

So I am spending 400 CPU minutes or 25% of CPU time… Is this the correct way to interpret this?


#6

Well already covered that… you can track it real-time in top, and/or over time via the information in the faq I linked you to.

You could write a shell script to process your resource reports, do a little arithmetic, email you the result, or add them to a text file, or whatever.

I don’t know of a single “command in SSH” that will gather for you all of what you want to see.

[quote]I downloaded some logs and I see the following:

Process CPU seconds user machine count average
php5.cgi 24828.5700 87.525% 103.452% 411053 0.060
php.cgi 3537.3700 12.470% 14.739% 112386 0.031
awk 0.7400 0.003% 0.003% 204 0.004
sort 0.3200 0.001% 0.001% 408 0.001
tail 0.2100 0.001% 0.001% 202 0.001
uniq 0.1100 0.000% 0.000% 204 0.001
suexec 0.0400 0.000% 0.000% 10 0.004
man 0.0400 0.000% 0.000% 2 0.020
cat 0.0300 0.000% 0.000% 7 0.004
ls 0.0300 0.000% 0.000% 13 0.002
troff 0.0200 0.000% 0.000% 1 0.020
bash 0.0100 0.000% 0.000% 1 0.010
nroff 0.0100 0.000% 0.000% 1 0.010
groff 0.0100 0.000% 0.000% 1 0.010
grotty 0.0100 0.000% 0.000% 1 0.010
sh 0.0100 0.000% 0.000% 3 0.003
find 0.0000 0.000% 0.000% 1 0.000
zsoelim 0.0000 0.000% 0.000% 1 0.000
sshd 0.0000 0.000% 0.000% 1 0.000
nroff 0.0000 0.000% 0.000% 1 0.000
tbl 0.0000 0.000% 0.000% 1 0.000
bash 0.0000 0.000% 0.000% 1 0.000
pager 0.0000 0.000% 0.000% 1 0.000
gzip 0.0000 0.000% 0.000% 2 0.000
locale 0.0000 0.000% 0.000% 1 0.000

Total: 28367.5300 100.000% 118.198% 524507
Average per day: 28367.5300 1 days
CPU percentage assumes 24000 cpu seconds per day total.[/quote]
That looks like a “resource report” to me (as mentioned in the faq).

Well, your arithmetic works out differently than mine. I get 28367.5300 CPU seconds divided by 60 (per the faq to get CPU minutes) = 472.792166667 CPU minutes (not 400) and I have no idea where you derived that 25% of CPU time number :wink:

You are pounding that machine really hard - I’m surprised they haven’t contacted you about that.

–rlparker


#7

Actually they did contact me several times. I was told to by CPU power in ps…

How much should I buy? How much did you buy?

Also I did some checking on looking at which part of my code consume the most CPU time. It’s when I curl someone’s else site…

You’re right it should be 470… Also 24*60/470 is around 33%… So I am using 33% of CPU power.

Is curl using that much CPU time? I thought it’ll just use bandwidth rather than CPU time?

How do I make curl more efficient?


#8

Well, then there you have it. They are essentially telling you your site is using too much CPU to remain on a shared server.

I don’t know. You haven’t shared the url of your site, and I don’t know what you are doing on the server or how you are doing it, so I would only be guessing. The nice thing about a PS server is you can manipulate it for optimum usage. If it were me doing it, I’d start out with as much as I could afford, then monitor it closely and reduce as much as I can without impacting the operation of my site. A day or two or three of having more resources than you need will not be that expensive, and I’d rather do that than have my site effected by having too little resources available.

That sounds like it makes sense. curling other sites can be CPU intensive.

Actually, I think you might be using a good deal more than that CPU percentage wise… that he math for CPU percentage is not a straight-forward computation (multi-core processors). The relevant point is that you need at least a PS server.

Well, you have the resource usage reports, and they show what they show.

I don’t know - I have not seen your site or your code. One obvious thing is to reduce its usage (either how much stuff you curl from other site or how often you do it).

–rlparker


#9

Very well, I will catch the URL I curl… That should greatly reduce CPU usage…


#10

Good luck with that. Hopefully, it will be enough, but be careful - if DreamHost has contacted you repeatedly about CPU usage, and that caching does not reduce it enough, they may feel they need to stake stronger measure.

Such measures could possibly include inhibiting the operation of your site. :wink:

–rlparker


#11

I now buy PS…

The thing is with this new PS thing I cannot know the load of my site… So I tried to make my site more efficient, but how do I know that my changes make my site more efficient?

There seems to be a graph that should show up in https://panel.dreamhost.com/index.cgi?tree=vserver.usage&

The graph is not drawn. I ask support and receive no reply.

Also I tried to split my domains into different user…

Usually I can ftp the log result. Now, since I moved to PS I can no longer do that?


#12

I don’t have a PS server, so I am not sure how your environment differs because of that. Hopefully someone will come along who knows about that and can help you more with it! :wink:

I think that all the resource tracking stuff we have previously discussed is also applicable to a PS server, but since you are not seeing it I am not sure.

It may be that if your PS server is “new” that data has not yet been generated for you.

–rlparker


#13

It looks like CPU issue has been solved…

However, now I am using too much memory…

Is there a PhP command that’ll allow us to see the amount of memory used or something?


#14

Since PHP is running as CGI, even if there was such a “magical command”, all it would likely do is show you that php.cgi is using it, which is less than helpful.

Tracking down exactly where PHP is using the most memory can be a pretty challenging process, but it can be done. To give you some idea of what’s involved , just take Google for a spin. You wll find quite a bit of information there, or you might just want to take a read through these:

http://www.ibm.com/developerworks/opensource/library/os-php-fastapps2/
http://phplens.com/lens/php-book/optimizing-debugging-php.php

As you can see, there is a lot invloved. :wink:

Fortunately, on the PS server, you are provided information to show you when you need to make more memory available for your server, so you can upgrade as needed while/until you track down what is using the most memory and take steps to reduce it.

You still haven’t revealed the url to your site, so the more experienced users here are unable to see things they might be able to instantly identify as “memory intensive” …

You also have not shared what software your site is running, or what it is doing (other than cURLing content from another site) …

At any rate, I know of no “php command” to profile your memory usage for you. You might find a skilled PHP programmer to help you “roll your own” testing routines to embedd in your script(s) at critical points, possibly using some existing php fucntions like:

http://us3.php.net/manual/en/function.memory-get-usage.php
http://us3.php.net/manual/en/function.memory-get-peak-usage.php

… and some of the example code listed in the “User Contributed Notes” following those manual sections, but it is not a trivial process to do that accurately or meaningfully.

What are you running that is generating so much CPU and memory consumption anyway?

–rlparker


#15

Okay…

Most of the time CPU usage is very low

Once in a while, CPU usage goes insanely high… My sites didn’t work and I got to restart the server…

When I run top it doesn’t look like I used a lot of memory or CPU, however, the graph says I am using a lot of memory…

The problem is even though CPU usage has on average went down by a factor of 10 and I have no freakin idea how I ended up using lot of memory given that TOP says I don’t… I am not allowed to move to normal server…

Be careful when you switch to PS… It maybe a one way road.

This is what TOP says:

Something is strange… It says that I used 350 MB… On the other hand, the sum of all sites I used is way less than 350 MB… Almost none of the process use more than 1 MB

Also why do I have so many sleeping process? Why the time mentioned in TOP are all less than 1 seconds while I see the same PID keeps existing for far longer than 1 seconds?

top - 20:45:21 up 2:36, 1 user, load average: 0.53, 2.23, 2.25
Tasks: 40 total, 2 running, 38 sleeping, 0 stopped, 0 zombie
Cpu(s): 13.0% us, 3.3% sy, 0.0% ni, 82.5% id, 0.5% wa, 0.2% hi, 0.5% si
Mem: 351232k total, 351232k used, 0k free, 0k buffers
Swap: 351232k total, 8792k used, 342440k free, 0k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM SWAP TIME CODE DATA COMMAND
24203 teguh123 16 0 20364 8876 4908 S 1.3 2.5 11m 0:00 6504 3232 php5.cgi
24212 teguh123 15 0 20364 8872 4904 S 1.7 2.5 11m 0:00 6504 3232 php5.cgi
31410 teguh123 15 0 2608 1384 1128 S 0.0 0.4 1224 0:00 592 228 bash
30100 nobody 18 0 2464 828 468 S 0.0 0.2 1636 0:00 428 580 proftpd
19649 dhapache 15 0 31956 17m 3924 S 0.3 5.2 13m 0:04 316 15m apache2-teguh12
21121 dhapache 15 0 31784 17m 3920 S 0.0 5.1 13m 0:03 316 15m apache2-teguh12
23078 dhapache 15 0 31904 17m 3864 S 0.0 5.1 13m 0:01 316 15m apache2-teguh12
23086 dhapache 15 0 32016 17m 3892 S 0.3 5.2 13m 0:01 316 15m apache2-teguh12
23419 dhapache 15 0 31872 17m 3860 S 0.3 5.1 13m 0:00 316 15m apache2-teguh12
23420 dhapache 15 0 31872 17m 3864 S 0.0 5.1 13m 0:00 316 15m apache2-teguh12
23423 dhapache 15 0 31856 17m 3860 S 0.0 5.1 13m 0:00 316 15m apache2-teguh12
23709 dhapache 15 0 31856 17m 3864 S 0.0 5.1 13m 0:00 316 15m apache2-teguh12
23837 dhapache 15 0 31744 17m 3852 S 0.0 5.1 13m 0:00 316 15m apache2-teguh12
23839 dhapache 15 0 31780 17m 4024 S 0.3 5.1 13m 0:00 316 15m apache2-teguh12
23840 dhapache 15 0 31744 17m 3864 S 0.0 5.1 13m 0:00 316 15m apache2-teguh12
23845 dhapache 15 0 31736 17m 3820 S 0.0 5.1 13m 0:00 316 15m apache2-teguh12
23846 dhapache 15 0 31752 17m 3888 S 0.0 5.1 13m 0:00 316 15m apache2-teguh12
24160 dhapache 15 0 31600 17m 3820 S 0.0 5.1 13m 0:00 316 15m apache2-teguh12
24161 dhapache 15 0 31792 17m 3856 S 0.0 5.1 13m 0:00 316 15m apache2-teguh12
24163 dhapache 18 0 31784 17m 3856 S 0.0 5.1 13m 0:00 316 15m apache2-teguh12
29952 root 15 0 31464 18m 5236 S 0.0 5.4 12m 21:43 316 14m apache2-teguh12
29955 dhapache 15 0 29700 14m 1196 S 0.0 4.3 14m 0:00 316 14m apache2-teguh12
29927 root 17 0 2736 1292 524 S 0.0 0.4 1444 0:00 276 876 syslog-ng
30090 root 15 0 3548 924 692 S 0.0 0.3 2624 0:08 276 456 sshd
31274 root 15 0 6124 1540 1244 S 0.0 0.4 4584 0:00 276 456 sshd
31404 teguh123 15 0 6108 1220 912 R 0.0 0.3 4888 0:00 276 456 sshd
31437 teguh123 15 0 2132 1068 836 R 0.0 0.3 1064 0:00 48 280 top
30083 postfix 15 0 3084 1220 1016 S 0.0 0.3 1864 0:00 40 264 qmgr
24099 postfix 24 0 3252 1532 1304 S 0.0 0.4 1720 0:00 36 436 local
1 root 15 0 1848 576 496 S 0.0 0.2 1272 75,23 32 280 init
24101 postfix 24 0 3108 1336 1132 S 0.0 0.4 1772 0:00 32 264 smtp
24097 postfix 24 0 3400 1660 1408 S 0.0 0.5 1740 0:00 28 436 cleanup
30106 root 18 0 1824 756 608 S 0.0 0.2 1068 0:13 28 248 cron
30080 root 23 0 3044 1156 980 S 0.0 0.3 1888 0:00 24 264 master
29970 root 15 0 1616 480 420 S 0.0 0.1 1136 0:00 20 232 inetd
24098 postfix 24 0 3232 1456 1244 S 0.0 0.4 1776 0:00 16 436 trivial-rewrite
24100 postfix 24 0 3232 1364 1176 S 0.0 0.4 1868 0:00 16 436 trivial-rewrite
30103 daemon 15 0 1768 388 308 S 0.0 0.1 1380 0:00 16 248 atd
30117 root 21 0 1572 448 388 S 0.0 0.1 1124 0:00 16 232 ud


#16

Sleeping processes are processes sitting there waiting in queue. Often they’ll be waiting for another process to complete then issue them instructions so they can continue with what they were up to.

That TIME value is how much CPU time a process has used rather than how long it’s existed.

Have you asked Support about those values?

Maximum Cash Discount on all plans during April with APRIL08


#17

TOP only display the top process… Is there a way to get TOP to display all running processes… I’d like to examine all those one by one…

Also, is there a way to decide which column to show up and which column should be started from the command line?


#18

top is quite configurable, both at runtime and with hotkeys while running.

http://man-wiki.net/index.php/1:top

Maximum Cash Discount on any plan with MAXCASH