My Client WordPress Website is not being crawled. Please Help

Hello all,

Here I’ve an crawling issue of my client WordPress site This site is hosted on Dream Host Server. This website contains default robots.txt which stop bots to crawl the site. “And that robots.txt file cannot be found in Source.” Now if we add our own “physical robots.txt” then it is visible on live. But it doesn’t make bots to crawl the site. It seems that it is due to system generated default robots.txt.

Please help me to resolve this.

default robots

I checked your site’s robots.txt and I cannot see any issue with it. How do you know that the site is not crawled?

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.