I would like to test a website without search engines crawling it. I’m guessing I set up a “test” directory with a robots.txt set to not crawl. Is that accurate?
Do I just create a test folder and upload the site to that?
you could put it in any directory you want to, and yes place the robots.txt file in the directory
** if you need help with robots.txt –
Promotional Code: [color=#CC0000]2DOMAINSFREE[/color] (2 Free Domain Names for the life of your hosting!)
For development purposes you could make a subdomian (eg. test.domain.com)
Put it in your test.domain.com folder.
Maximum Cash Discount on any plan with MAXCASH
If I was really worried about robots crawling a site, I’d go with a really obscure subdomain, and maybe even password protect the site.
There’s no guarantee that an unscrupulous crawler will respect the robots.txt file