Thrasher hardware change ... 30 hours off line


#1

DreamHost is fixing a problem with the thrashers server and is doing a hardware swap. It’s been down for over 30 hours on some of my sites. How long should this take? DreamHost has not posted any update to those of us on thrasher for over 24 hours (see below).

! Important Information: Shared server ‘thrashers’ being moved to new hardware (updated 1 day 0 hours ago)
Unfortunately, the state of the backup server is degraded to an extent that we won’t be able to see much improvement during the restore process. We are restoring your data as quickly as possible while avoiding additional stress to the server itself, to avoid more critical service interruption. We are also looking into alternate options to continue the restore on a more stable ground. Please note, as soon as your users are restored, you’re switched to the local drive, and will no longer be affected by the slowness of the connection to the backup server. We apologize for the inconvenience this is causing. We will update this notice later today or sooner if there is a change in status which necessitates additional notification.


#2

I’m also on thrashers. I’ve always been satisfied with DreamHost but this outage is getting pretty long and all the status messages have been quite vague on how long they expect it to continue.


#3

Me too and I have been a DreamHost user for a long time. What bothers me is the lack of communication - this MAJOR DOWNTIME (for us on thrashers) is not even reported on the status page. No communication other that “all hell broke loose, we may be working on it … check back tomorrow.” I don’t rely on my websites for income, but if I did I would be asking for recommendations for other services that don’t shut down for 40+ hours with little or no information. FYI I am trying to upload to one of my sites … I’m trying to move 4.2MByte (not much). It looks like it will take over 9 hours to move that data to the server. No wonder my sites are very, very slow to respond.


#4

I’m another longtime Dreamhost customer on Thrashers who isn’t impressed with this at all. Normally, Dreamhost is very good at answering questions quickly.

I do have business sites on the server and though clients were understanding initially, I don’t think that will continue to be the case…

I may have to find a new host if there’s no movement on this today.


#5

When I asked on twitter (@DreamHostCare) about downtime ETA for Thrashers I got a fairly canned no-info response.

Other tweets there seem to suggest a one week downtime, but just a rumor at this point.


#6

I’ve resisted replying here because the information posted above doesn’t have the word RAID in it. However if this is a RAID controller issue, that used to cause disruption for about 2 weeks (search the word RAID in this forum). Recently there was a post that looked closer to 3 weeks out which made me wonder if RAID controller issues now take longer or the poster was an isolated case.


#7

Yes, it is listed as a RAID controller problem. This doesn’t bode well. I was not aware that the RAID tech had such a high-downtime risk.


#8

My clients website “www.mooneypilots.com” on server “thrashers” has not been able to be updated now for several days. My personal website on server “bulger” is working just fine. Can you just transfer my clients website from “thrashers” to “bulger”?


#9

I finally got a reply from my support ticket - 48 hours later. It basically says:

“We are continuing to work on it. The goes in thingie is not very fast and the goes out thingie is just as slow. NO ETA for completion. Very sorry blah blah blah.”

Very frustrating. I wonder how many users are on thrasher? How big is the RAID?

Does anyone have a recommendation of another provider that has better service? I generally like DreamHost … I’ve been using their service for 9 years! I really don’t want to change … but really! I suspect most of us on thrasher have been with DreamHost for a long time.

Warning to the rest of you - DreamHost seems to be fragile and not prepared for (some) hardware failures.

I will be interested to see how they deal with us after this is all fixed up!


#10

they don’t… there is plenty of prior raid failures. Just use the search function.

The ironic part is that the purpose of using RAID (Redundant Array of Independent Disks) technology is to avoid down time. RAID controller failures apparently crash the entire array tho. Literally at dreamhost the fix is 2+ weeks of meaningless replies from support as everyone waits for the drives to rebuild. Supposedly during this time they mount the backup as well, and support will say they checked your sites and they were operational. After about 2 weeks the whole thing fades to background for the currently effected customer group. And I’m positive they lose a customer or 2 each time… but that’s not enough to make a dent, and after a month or 3 someplace else probably half the customers are back at dreamhost.


#11

LakeRat - thank you! I did the suggested research and it does seem like it takes a random amount of time to get RAID failures fixed. I would rather know what I’m facing than just hope it will fix in the next hour…

That being said, it looks like thrasher is back up and running for my FTP accounts! Does not look like any lost data. Seems to be normal again. So it took 3 or 4 days for one user. I suspect it is one user at a time. Good luck to the rest of us on thrasher.


#12

LakeRat

Presumably you are speaking in units of a thousand.

If so, I agree. Losing one or two thousand customers every time this happens may not be enough to make a dent, given that they have about 380,000 customers.


#13

No I’m not. There is nowhere near one or two thousand customers impacted from a RAID card failure. I have no idea how many actual accounts are affected when a single servers RAID card fails, but in the grand scheme of things it’s a very small percentage of a number like 380,000. When they used to use dreamhoststatus.com differently and alot of server names popped thru there, I was always amazed at the number unique names. I would imagine the users on a server affected by RAID card failure fall into 3 categories: “casual user… oh well i’ll just let this run it’s course… that’s easiest”, “users that are mad enough they want to move, but don’t have the technical expertise needed to actually move the site(s)… so they too wait it out”, and finally “the web designer reselling that’s smart with servers and can move somewhere else quickly and efficiently” — the only guy to worry about is category 3, and he’s a pretty small percentage of the single failed servers population. AND I bet half of those that leave don’t close there account and move the sites back within 3 months. Net loss to dreamhost for a RAID card failure is nothing more than normal churn.