Google Bot, Crawler SEO


#1

Hello, I just signed up with a DreamHost account!

I’m a little concerned though, I recently read that DreamHost has been blocking the Google Bots? Is this standard on all shared hosting plans or only when it becomes a problem.

I’m hoping my site will eventually get crawled by the search engines for keywords that I’m hoping will attract my target audience.

Can anyone comment on DreamHost stopping search engines from crawling their sites on shared hosting plans?

http://www.unofficialseoblog.com/2007/05/18/big-brother-dreamhost-demands-obstructing-google-bot/


#2

Welcome to Dreamhost! :slight_smile:

It is not “standard” practice at all, and has only been used in unique circumstances when the bot was hammering a server into submission.

There were actually very relevant and insightful comment from Dreamhost staff already out there in response to that story…I’ll have to google to find them now. In essence, if a bot tuns amok, blocking it rather than taking the site down, seems to be a more desirable approach to my way of thinking. The main point is, this is a rare occurrence, and understandably, the SEO pundits and the affected user made a much bigger deal of it than it really is. :wink:

There is also discussion of this, and a response from DH in this DH forum thread from earlier in the week (the link is to Jeff from DH’s comments - you might want to read the whole thread for more insight)

–rlparker


#3

Thanks, I figured this was the case but just wanted to make sure it wasn’t a policy to block the crawlers. I did a search on this topic before posting and didn’t find any information - probably because it’s a mute point…

Regards, Joe


#4

No problem…I found one of the responses from DH in this DH Forum post.

–rlparker


#5

Are you willing to put in the time to make sure that your site can be crawled effectively without the bot getting stuck in circular links or eating up server resources?

If the answer is no, then I would be a little worried.

If the answer is yes, I wouldn’t be worried. In fact, I’d be pleased that DreamHost is being proactive about protecting me from selfish numbskulls who don’t care about anyone else they’re sharing their server with and just want to slap together cheap poorly coded websites and monetize them.

------ (That’s it for my reply to the OP. The rest of this is a rant.)

Yes, I’m aware that I’m departing from my usual sympathetic and politically correct postings, but I really do want to make sure that DreamHost doesn’t get burned for trying to help out their core user base - us small-timers who just want our personal websites to run on rock solid stable web servers with consistent performance. I think the same goes for the more upscale users as well - it is in everyone’s best interest to learn how to create efficient websites and have them crawled efficiently in order to keep costs down and service levels up.

My personal beef? The dumbass who had some script that ate up all the server CPU and took my server down for four hours two weeks ago. I applaud DreamHost for doing whatever it has to in order to keep my server up and thus keep my sites up.

Sigh, I would like to apologize for the rant. It seems that all I see here lately is people complaining because they were inconvenienced by DreamHost acting to make sure that these people’s scripts didn’t compromise the ability other users to access shared server resources. And these complainers do not seem to exhibit any remorse for the tens of people they have inconvenienced by making the server unusable.

It’s presumptuous of me to say this, but to those considering signing up for DreamHost: If you’re going to be a bad neighbor, then please don’t move into the neighborhood. If you are going to be a good neighbor, then please please please join us. We need you and we need more people like you!

----- (back to the OP)
You sound like a polite, concerned, thoughtful person, so I’m sure you’ll fall into the “good neighbor” category!

Free unique IP and $67 off with promo code [color=#CC0000]FLENSFREEIP67[/color] or use [color=#CC0000]FLENS97[/color] for $97 off. Click here for more options


#6

Speaking of bad neighbors, if you can get my neighbor nextdoor to mow his grass (which is a ratty jungle) I’ll give you a superhero reward!


#7

Buy him a pet goat?

Free unique IP and $67 off with promo code [color=#CC0000]FLENSFREEIP67[/color] or use [color=#CC0000]FLENS97[/color] for $97 off. Click here for more options