Dumb notes here…
Wouldn’t it be helpful if DH put a folder explorer feature in the new panel page for robots so that we can easily find/select directories that we don’t want indexed?
Since contra-bot-botting is such a hot topic these days… Seems to me that robots.txt should really only include those directories that we know are already indexed that we do not want indexed. It’s counter-productive to list directories in robot.txt that you don’t want indexed because legitimate bots and ne’er-do-wells are going to read the file and from that point on know exactly what you don’t want anyone else to know about. In other words, do not list [color=#0000CC]/secret[/color] in your robots.txt file! Errr, right?