Recently I have seen someone mention a file robot.txt, related to the "google bot". The web crawler,..apparently it needs to find the file "robot.txt" to function correctly ? Can anyone explain more on this, where should that file be, ? What should be in it ?
Thanks, from Garry