Robots.txt


#1

I looked back and searched, but I still haven’t really found the answer to my question, so I’ll ask now!

It’s about robots.txt files.

Okay.

Where do I put it? Say I am using the WebFTP, I would put it in the
root/domain.tld folder?

And do I have to create one for each subdomain, or would the one main one in the main domain be enough?


#2

http://www.google.com/search?hl=en&q=where+to+put+robots.txt

Yes, yes, no.

----------
[color=#00CC00]ANONYMOUS0 => Max Discount[/color], [color=#6600CC]ANONYMOUS2 => Max Discount less $1[/color] or Cut the code, [color=#CC0000]it’s their birthday:[/color] [color=#00CC00]9999 => $99.99 Off[/color]


#3

[quote]Where do I put it? Say I am using the WebFTP, I would put it in the
root/domain.tld folder?[/quote]
Yes, the robots.txt file goes into the root of the web-directory for the domain (or sub-domain).

You will need to place the file in the web-directories of the parent domain and each sub-domain.

Mark


Save [color=#CC0000]$50[/color] on DreamHost plans using [color=#CC0000]PRICESLASH[/color] promo code (Click for DreamHost promo code details)


#4

Wow - two answers in as many minutes… well done guys.

I had another question regarding the robots.txt files. Some SEO “experts” seem to be espousing the merits of using robots.txt even if there aren’t areas that you need to restrict because they increase SE traffic. Is there any weight behind this statement or is it just mumbo-jumbo?

Thanks,
-James

http://www.jconnors.net - City Streets


#5

I can’t say I’ve noticed a difference in bot traffic between sites that have a robots.txt and those that don’t. You can use the Crawl-delay parameter to limit bot traffic, but I know of no robots.txt parameters that will actually increase bot traffic.

I’ve found that the best way to increase the frequency of crawling on my sites, at least from Google, is to use Google Sitemaps and (of-course) to link to the site from as many places as possible.

Mark


Save [color=#CC0000]$50[/color] on DreamHost plans using [color=#CC0000]PRICESLASH[/color] promo code (Click for DreamHost promo code details)


#6

I don’t want to limit or delay them. It was just interesting and a little too good to be true when some of these people were saying “oh my traffic went up 16% as soon as I put up the robots.txt”

At this point, I’d just be happy if googlebot came back to my site. It hasn’t visited in a couple weeks :frowning:

http://www.jconnors.net - City Streets


#7

Is this the same site as linked in your signature? If so, then I don’t think you’ll have too long to wait. In my experience, links in this forum tend to get the attention of the Googlebot. :wink:

Mark


Save [color=#CC0000]$50[/color] on DreamHost plans using [color=#CC0000]PRICESLASH[/color] promo code (Click for DreamHost promo code details)


#8

It’s one in the same - webmaster tools shows the 19th of april the last time it crawled my baby.

Edit: I just looked at the stats on the webserver and it shows that googlebot had been there in the last seven days… yet webmaster tools says a couple weeks - SO CONFUSED!!

http://www.jconnors.net - City Streets


#9

The date shown in webmaster tools seems to be severely lagged, webmaster tools currently shows the following for one of my sites;

“Googlebot last successfully accessed your home page on Mar 30, 2007.”

But the site in question is visited many times each day by the Googlebot, as verified by the TraceWatch realtime stats package that I have installed on the site.

Mark


Save [color=#CC0000]$50[/color] on DreamHost plans using [color=#CC0000]PRICESLASH[/color] promo code (Click for DreamHost promo code details)


#10

Do all of your pages come up in a site:yourdomain.tld search? Only the pages that were in existence on the 19th come up for me :frowning:

EDIT: Well here’s one for you - I did the site:jconnors.net search and it seems that Gbot has cached some new pages as late as the 4th of May. Maybe it only changes the name if your homepage vs. blog pages changes/updates?

http://www.jconnors.net - City Streets


#11

Yes. Well, all except the pages added very recently. I’ve found that it generally takes around a week for newly added pages to show up in the Google index.

Mark


Save [color=#CC0000]$50[/color] on DreamHost plans using [color=#CC0000]PRICESLASH[/color] promo code (Click for DreamHost promo code details)