Interpreting Analog Stats


#1

How do I read the standard Analog stats that are provided to determine if google (or other) bots are a significant source of CPU consumption?

I have a static site that’s typically the first google result for my key phrase. I’m reasonably certain the vast majority of visitors arrive there by google searches.

However I also have a forum and gallery (these two have AdSense on them). I see a google bot on the forum visitor list pretty much constantly, and the google index of posts seems to stay very up-to-date. I hear people talk about their sites being ‘pounded’ by bots - how would I determine that by reading the analog reports? How can I tell the difference between ordinary traffic arriving THROUGH google, vs cycles churned BY google?

Sorry if this is a really dumb question.


#2

The “Browser Reports” should list the Googlebot if it is a significant source of requests, otherwise you might want to use different software to analyze the log files yourself for the Googlebot request.

Additionally if you don’t want the Googlebot to index the forum or other part of the site, use a robots.txt file to restrict access; see http://www.robotstxt.org/wc/exclusion.html

:cool: Perl / MySQL / HTML+CSS


#3

Thanks Atropos7. Looking at 12,004 page requests so far today, 1,241 of them are Mediapartners-Google and 659 of them msnbot. That’s about 10%. I dunno, that doesn’t seem unreasonable.

If I look at the current monthly (183,110 pages) neither of them even show up in the top 5.

I have msnbot on a crawl-delay and have a large number of reportedly bad bots excluded in robots.txt. I suspect banning googlebot would not help my AdSense relevancy much :slight_smile:

Thanks for the tip. Now I know where to look!