Robots.txt


#1

I’d like to make it so DreamHost does not automatically insert the NoRobots HTML tag into my sites. How do I do this?

I am not a total n00b to the net, but I am new to DreamHost (I love it). Thanks for help.


#2

I did not know they did this. Is it a new feature?
So they have inserted a NoRobots HTML tag into your site?

Can you post the line of HTML containing this ‘tag’ please?

There is a manual option under goodies>block spiders to add/remove a robots.txt file if that is what you are asking about.


Norm

Opinions are my own views, not DreamHosts’.
I am NOT a DreamHost employee OK! :@

Act on my advice at your own risk!


#3
// This was my tag I inserted into the head ... // This is theirs, I think

I deliberately added the robots follow meta tag. As you can see, the meta tag is inserted on my page automatically to appear right before the closing head tag. I hope I am not being a complete dunderhead about this. It is a Tumblr blog, but I don’t think that has anything to do with it. See for yourself at http://internetjay.com

I tried the Goodies the first time, but there doesn’t appear to be anything that says it will turn OFF blocking robots.


#4

So you have a domain name which has an A record pointing to a tumblr log at 72.32.231.8?

I just wonder then if it is something that a tumblr theme is doing.
I looked at the tumblr log site and see that you can add a custom html theme via the dashboard but you need to use special variables and blocks.

I would try a quick message to tumblr support to see if it is their system that is changing your meta tags.

I have put your robots meta name tag into a DreamHost HTML page and it has not changed yet.


Norm

Opinions are my own views, not DreamHosts’.
I am NOT a DreamHost employee OK! :@

Act on my advice at your own risk!


#5

I am sorry. This was not DreamHost’s fault, but a custom setting inside Tumblr to allow/disallow search engines from indexing your site.