Page not indexable b/c of robot.txt


#1

I recently changed to a new theme on my wordpress website/blog. I was ranking really well, but now I am not ranking at all, and I am receiving a message…
“Page is not indexable because of robot.txt” from my SEO Doctor

How can I make my page indexable again?

Thanks!
Sarah


#2

What does your robots.txt file look like?

Jw

Basic one:

For more information about the robots.txt standard, see:

http://www.robotstxt.org/orig.html

For syntax checking, see:

http://www.sxw.org.uk/computing/robots/check.html

User-agent: *
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /wp-content/cache/
Disallow: /wp-content/themes/
Allow: /wp-content/uploads/

http://www.askapache.com/seo/updated-robotstxt-for-wordpress.html

Jw


#3

where do i find that? I’m sorry I’m totally lost.
sarah


#4

Okay, so i tried to fix the problem myself… bad idea… i cant access my site at all anymore :o( i replaced the robox.txt under the seo plugin from

BEGIN WordPress

RewriteEngine On RewriteBase / RewriteRule ^index\.php$ - [L] RewriteCond %{REQUEST_FILENAME} !-f RewriteCond %{REQUEST_FILENAME} !-d RewriteRule . /index.php [L]

END WordPress

to

User-agent: *
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /wp-content/cache/
Disallow: /wp-content/themes/
Allow: /wp-content/uploads/

Help!?!!! Please, i really need my site up.


#5

That’s not your robots.txt file, that’s the .htaccess file…

Restore that back to it’s original form, you’ll probably have to sFTP the original one since you’re probably getting a 500 error right?

Jw


#6

robots.txt and .htaccess are not the same. You put the code for robots.txt. in .htaccess.


#7

Oh, gotcha, so how do i access the robots.txt. and what do I canhge it to? I’m super nervous now about all of this…


#8

#9

Okay, so after looking online i see that I am looking for the text…

User-agent: *
Disallow: /

But, I don’t see it in any of my templates or the style sheet


#10

Hi there. It’s possible you do not have a robots.txt page.

Just create one with a plain text editor and copy that suggested text into it.
FTP to your site and upload the file (you can also create this file by logging the shell and using vi)
That file should go into your domain root folder.

See if that straightens things out.

Also:

You mention this issue only occurring after a template change. Does this template have an admin panel associated with it? It’s possible you need to set up the robot/indexing directives from within the template panel/WP dashboard.

See if the template developer offers some support.

One last thing:

Since you messed around with your htaccess file, in your WP dashboard, go to Settings > Permalinks and re-save/refresh.


#11

I believe you don’t have the robots.txt file. You must create it.