.htaccess and robots.txt/sitemaps, etc



Okay, I have a custom .htaccess file that redirects my wiki to the /article/path. That works perfectly, and we’ve figured out how to let the stats pages still work independently.

But I can’t figure out how to allow robots.txt, sitemap.xml, and googleverifypage.html to load outside of the htaccess rules. Everything I’ve found so far doesn’t work.

Any ideas? :frowning:


It might help to see your .htaccess redirect formula.



one would think that, wouldn’t they? :slight_smile:

The #robots.txt lines are not working. I pulled those from apache, but alas. :frowning:


I think robots.txt should just be in the root of the site and there should be no need to have it in .htaccess.

You should put it in the main folder so it can be found at www.mysite.com/robots.txt you would then put something along the lines of

Disallow: /wiki/page or Allow: /wiki/page

$97 DISCOUNT with [color=#CC0000]DISCOM97[/color]
More codes


if i’m remembering correctly, RewriteCond only applies to the next RewriteRule, which is your awstats workaround. your robots.txt requests are getting rewritten by your mediawiki beautifying urls.

track7 - my dream-hosted site


hey Monkeyboy,

That’s where it is. But since the rewrite grabs every request not defined by the htaccess, and applies the beautifying, it tries to see robots.txt as a wiki entry. :frowning:


[quote]#robots.txt workaround
RewriteCond %{REQUEST_URI} !^/error/(40(1|3|4)|500).html
RewriteCond %{REQUEST_URI} !^/favicon.ico
RewriteCond %{REQUEST_URI} !^/robots.txt
Maybe this is not working because you have set up some conditions, but never setup a rewrite rule to be applied if the conditions are met? :wink:

Note how your working sections that are working are setup.



hey RL,

I tried that. I added the rule + [L] (last) and that failed. Tried [U], which checks if the above are correct, otherwise moves on, and that failed.

This is why everyone in the world suggests putting your wiki is a subdir. heh.

I’m going to move the wiki to a subdir, remap .htaccess to beautify, and leave it at that. :slight_smile:


Sorry that didn’t work - I freely admit that, for me, sometimes it has taken the mumbling of arcane incantations over the entrails of small beasts to get mod_rewrite to work as I expect it to. That stuff can be pure voodoo-black-magic as far as I’m concerned.

The subdir might, indeed by the way to go! :wink:



try getting rid of the negation (!) on your RewriteCond lines, then putting this RewriteRule after:

RewriteRule .* - [L]

that should rewrite it to itself and stop processing rules. my wiki at wiki.track7.org doesn’t show any sub directories so it can definitely be done! here’s what i have in .htaccess for that site – note the main page of the wiki is “Contents”

[code]RewriteEngine On

anything that exists should be left alone

RewriteCond %{REQUEST_FILENAME} -f
RewriteRule .* - [L]

anything that contains a dot without a colon should be left alone

RewriteRule ^[^:]*. - [L]

anything that contains a slash without a colon should be left alone

RewriteRule ^[^:]*/ - [L]

redirect to main page

RewriteRule ^/*$ /wiki/index.php?title=Contents [L,QSA]

anything else is to be treated as a title

RewriteRule ^(.+)$ /wiki/index.php?title=$1 [L,QSA][/code]track7 - my dream-hosted site