Robots.txt


#1

I understand what this is (sorta), but where is it? Or if i have to make one, where do i stick it?

[quote]_>
[/quote]

And how would i make one?


#2

A robots.txt file is just a regular text file containing specially formatted entries. You create a robots.txt file with any text editor (not a wordprocessor),and upload it into your website’s “base” directory (by default on Dreamhost, that directory is generally /home/yourusername/domain.tld). It goes in the same directory as your primary index.html (or other “default”) file.

–rlparker


#3

These sites can get you pointed in the right direction:

robotstxt.org

Wikipedia


:stuck_out_tongue: Save up to $96 at Dreamhost with ALMOST97 promo code (I get $1).
Or save $97 with THEFULL97.


#4

And you may want to refer to this article http://www.outfront.net/tutorials_02/adv_tech/robots.htm It explains how to create robots.txt file step by step.

Save [color=#CC0000]$97[/color] (max discount) on dreamhost plans by using promo code: [color=#CC0000]97CRAZY[/color].


#5

Here is the contents of my robots.txt file: http://tzfiles.com/robots.txt

I don’t really know what the first line says, but the second line tells robot such as the google bot, or the yahoo bot, that they can index any site in my directory.


#6

[color=#0000CC]User-agent: *[/color]

The ‘*’ in the first line is a wildcard character, this line basically states that the rule below it applies to all user-agents.

Mark


Save [color=#CC0000]$50[/color] on DreamHost plans using [color=#CC0000]PRICESLASH[/color] promo code (Click for DreamHost promo code details)