I was just notified by DH support that we are being watched for excessive usage - about 100 cpu minutes a day or 28% of the server. I cannot get the log details yet, but an working on it.
I have a PHPBB and PostNuke CMS running. Could these together be causing this much cpu time?
TIA,
Bill
depends on how many users are hitting it/them throughout the day
we just got http://www.gurusnetwork.com off of an eval server after the logs showed mostly search engine robots hammering us. We looked at robots.txt directives and implemented those and personally contacted the other engines that don’t follow the crawl-delay directive (Google!) and asked them to take it easy.
This was the majority of the problem, Google and Yahoo Slurp were the biggest offenders, but there are a lot of generic site downloading bots that hit the GN pretty hard at times, takes a diligent person to keep on top of that stuff and block offending IPs.
blocking the search engines had an immediate effect on page rank in the search engines too, so we gotta either find a happy medium or go dedicated
any previous mention of “us” almost certainly pertains to a diligent creature name of Emperor thereabouts
Being on an eval server now, I can relate! Any chance you might consider starting a wiki topic with ‘suggested’ IPs and techniques - i.e., a shareable robots.txt and/or .htaccess lines? I just slapped a big honkin’ list of so-called ‘evil’ robots I found somewhere into robots.txt out of paranoia. I thought crawl-delay was just an MSNbot thing - even though it ought to be a standard, I think. I sure did see MSNbot on my (IPS as it happens) forum constantly - Inktomi camps out for days on end too. I’ve been proud of having our club site be the number 1 Google result for my dog breed, but I guess there is a dark side. There is no way on Earth that our dog club is going to pay for a dedicated server to support ‘community’ - my club as a whole is still pretty much in the ‘home page’ mentality. My choice will be to have, or not-have, forums. Anyway, I bet a lot of folks could benefit from some specific damage control tips. I want to be a good neighbor because I want to have good neighbors.
I’ll put the wiki on my todo list
which is rather long atm =\
edit after looking at the wiki this just fell way down the list, like I need to learn another lotta markup language and organization scheme 
thought I hated wikis, now I know! I’ll try to get the info into here soon then anyone that is wiki savvy can add it.