IMPORTANT: Dreamhost is down. Mail systems do not work


#1

Hello,

DreamHost Team.

Your servers are down and mail systems does not work. I cannot access panel.dreamhost.com or www.dreamhost.com. I cannot fill a form to support team as well. I sent you an email, but I am not sure if you will receive it, this is why i create this topic.

Please let us know what happens with the servers, mail systems, etc.

I will be very happy if you bring up your services as soon as possible. It is an emergency situation.

Regards,
anebi


#2

Same here. No access to my sites, everyone complaining at me and www.dreamhost.com and therefore the control panels are down along with emails. joy


#3

Same here. Websites work fine now, but mail is down, dreamhost don’t say nothing, dreamhoststatus.com no updates! panel.dreamhost.com!
Anybody say nothing!


#4

Hmmm… It gets really frustrating sometimes. I don’t understand how they can take the whole system down like this. What happened to redundancy?


#5

Yes, this is right. I expected such a big company to have at least some basic failover scenario for such situations.


#6

I’m not personally cognizant of their network topology, but if Amazon, Apple and Google can have occasional server hiccups, you have to expect that a much smaller company, such as Dreamhost, will as well. It’s an imperfect world out there when it comes to servers. The cloud is still a work in progress.

Peace,
Gene


#7

Agreed. However, this was a planned outage that lasted longer than initially planned.

The passive notification system for planned outages is a massive fail. We have to remember to manually check dreamhoststatus.com, and we have to remember to do it regularly because the notices are often posted only a day or two early.
For example:
–last nights Maintenance Announcement was posted was posted originally on the 18th, just 55 hours before the start of the event.
–the recent OS upgrade outage announcement for my server was posted just 5 hours before it was to begin.

Many of the comments over on the dreamhoststatus.com thread from last nights outage suggest email announcements of planned outages. I think there should also be a way to see planned outages in the panel, complete with the Alert Tab turning red or something if there is active information about an upcoming planned event.[hr]
As an after thought, that panel addition could actually driven with an XML or other type of feed from dreamhoststatus.com and be filtered down to just what effects your account.


#8

It’s also true that “planned outages” almost always take longer than predicted. That’s an unfortunate part of the process.

I have nothing to say about whether they could or should have posted this information farther in advance, though.


#9

I’m not complaining because it took longer than expected, or because even because it had to happen. My complaint is that [size=large]the notification system is PASSIVE[/size]. There should be a way for us to learn about these things before we check dreamhoststatus.com because things are not working right, and we shouldn’t have to remember to check dreamhoststatus.com every few hours to see if something is coming up.


#10

I agree. I had no idea what was going on or anything about dreamhoststatus.com. Thank you for the information so I can stop panicking!


#11

Well, Namecheap – which also has a hosting service – does let you subscribe to such notices. So you do know in advance when they need to fix something.

Peace,
Gene


#12

You can subscribe to dreamhoststatus.com using RSS.


#13

But just getting email notices is better. :slight_smile:

Peace,
Gene


#14

Yeah, but it’s not that simple. DH actively promotes a something that has not worked for a long time and when you ask support about it, they all know it doesn’t work. You go to the panel and it says:

New! Get a CUSTOM feed of our DreamHost status announcements of JUST the issues that affect you: Subscribe to RSS DreamHost status RSS feed for Account #xxxxxx

So yes, there is an RSS feed, but how effective is it when 99% of it is irrelevant? I have questioned support about this before, and they said something like ‘yeah, we know it doesn’t work, sorry’. And I’ve told them ‘then take it down. Stop promoting something that doesn’t work’. But it’s still there, tempting people into thinking that they can get customised warnings about issues that relate to them, but it doesn’t work.


#15

Subscribing to passive notification via a passive service (even if it did work right) is still not [size=large]ACTIVE[/size] notification.


#16

It’s interesting that there have been two blog updates since the outage, but no word at all about it. Is this just going to be swept under the rug? I think one of the reasons why people like DH is because of their flexibility (in terms of generous resource limits and a willingness to work with users who use too much) and their honesty (in terms of admitting when they screw up and trying to make sure it doesn’t happen again)

Yes, catastrophes happen, but with the new website and relatively meaningless blog posts that have been coming out lately (such as the Amazon post, what was that all about?), I’m starting to wonder if the personality of DH is shifting.


#17

Good points bobcat. I’m also reasonably certain that one of the status updates that morning promised a postmortem of the incident, however after reading your post above I went back to find that update and see if dreamhost had said anything additional and I can’t even find update that contained something about a postmortem explanation. :wink:

Dreamhost should study the concept of maintenance windows. Specifically, a pre-announced schedule that maintenance might occur, such as Thursday’s from midnight to 4AM PST. (That would actually be overnight on Wednesday night.)

The problem is saying Thursday’s from midnight to 4AM PST is not a good time for everyone worldwide, since at midnight PST it is already 8AM in the UK and even later in the day in Australia/New Zealand.

The solution is to have 3 scheduled Server maintenance windows, one for the Americas (North and South), one for western European countries, and one for the remaining Eastern European/Pac-Rim countries; and have one Network maintenance window that might effect everyone if it’s used. (Just because there is an announced pre-determined maintenance window doesn’t mean it has to be used, but if there IS any routine maintenance to do that is when it will be done.)

Customers are most likely not now assigned to a server by region of the world, but that doesn’t mean they couldn’t/shouldn’t be, and it might take awhile to move accounts around to make it so, but it could be done.

I have previous experience in the industry with a company offering an enterprise solution worldwide and this is how scheduled maintenance was handled.

This is good for all, including Dreamhost, because this can be written into the TOS and whatever windows apply to any specific account can be included in the “Account Status” section of the panel as regularly scheduled maintenance windows.


#18

The geography-based windows are a good idea, but in my case, I’m in the Australia/NZ area physically for the time being, but most of the visitors to my site are in the US, India, and Japan. So, I’m not sure where I would fit in. If something must go down, then a weekend would be better than a weekday. Late Friday night in Cali is the weekend everywhere else.


#19

and there are customers that will say weekends are the most important time for their sites to absolutely be up. One example I can think of right off are any sports related sites, that’s exactly when their peak traffic occurs.

There is no one good time for all. Nor will everyone be happy being assigned to a maintenance window based on THEIR geography because even tho they are located in country A most of their traffic may come from country B in another region.

Perhaps the customer should get the chance to pick between 3 or more choices outlined above for the server group maintenance window. For the entire network maintenance window, since it affects everyone, they would have to assign that to the time that they think worldwide traffic is usually the lowest (if there is such a thing). Hopefully the need to use this window would be infrequent.

In addition to Dreamhost being able to tell its customers when to expect regular maintenance to occur, Dreamhost customers can tell THERE customers the same thing. “The regular maintenance windows are…, they may not always be used but that is when your site might go offline for routine server/network maintenance.” Being in pro-active notification mode saves a lot of emails and angry posts.

This post on Dreamhoststatus.com was made just 5 hours before the scheduled maintenance was to begin. I only found it because I went to find out why the server was down. How could anyone be expected to find that and notify anyone they need to notify?


My apologies bobocat, I’ve pulled this in another direction from where you were going when you reactivated the thread.

I think we need to take it back bobocat’s direction and talk about whether the outage is going to be swept under the rug and whether the personality of DH is shifting.

FWIW: I don’t think Dreamhost has the same personality today that it did when I signed on (close to 4 years). I had been looking for another host for awhile, I read 2 years worth of dreamhost newsletters one night and signed up the same night. I never have moved my paying customers here as was the plan at one point (I do have a number of personal and test sites here). I’m currently working on a project that is going to require a dedicated server in a month or two when it goes live, I haven’t yet decided if I’ll choose dreamhost, in many ways it’s my first choice but issues like these make me wonder if I should look for other solutions.


#20

[quote=“LakeRat, post:19, topic:56262”]
I’m currently working on a project that is going to require a dedicated server in a month or two when it goes live, I haven’t yet decided if I’ll choose dreamhost, in many ways it’s my first choice but issues like these make me wonder if I should look for other solutions.[/quote]

That was sort of the basis for my comment earlier about not having redundant systems. Ok, in my case, my site doesn’t, and never will, make any money nor will it ever be popular enough to justify a VPS or dedicated server, so I can live with shared hosting and, despite the inconvenience and the 100% uptime guarantee, I can live with some downtime. That’s life. But DH really pushes VPS and now dedicated servers. That’s really money for big sites. I’m a little disappointed by the downtime fiasco, but if I were paying $50+ per month, or however expensive a dedicated server is per month, I would expect a more robust system.

Some have mentioned failures at Amazon, Apple, and Google. Of the cases that I’m aware of, never did the entire system go down. Some services went down. I recall with the Amazon case, the sites that were affected there were the ones which tried to save some money by only using one location rather than paying the extra money for multiple locations. You get what you pay for. I’m not sure how expensive DH dedicated servers are compared with Amazon, but if they are close, then they should have a more robust system.

The outages at Google that I know of generally revolve around certain services such as Gmail. I don’t pay for Gmail, so fine, I get what I pay for. But they do have paying customers and they rightly should have lost a lot of business when their systems went down.

Ok, I’m going random here. But the point is, DH has already had a fair share of complete network meltdowns and each time has promised that a system wide failure won’t happen again. They’ve been honest about that in the past, and very apologetic. There was even a case a few months ago when their router config files kept disappearing or something. They fessed up to it and offered compensation for those affected. I didn’t even take them up on the offer because I didn’t notice at the time and looking at my logs, I didn’t see a big hole in the traffic volumes.

I don’t know why this time is different. Whey are they posting all of these ‘meet the customers’ blogs and completely ignoring a huge, system-wide failure? What’s up DH? Don’t pull an Apple on us and pretend that there’s a design flaw in the system.