Current outage - all sites down 24 hours +


#1

Yesterday I got the message there was a hardware problem with the vps server my sites are on. It said the sites would be working while they did the hardware change. Contact them immediately if I had problems.

I did, and my sites have been down over 24 hours now and no reply from support.

Here’s the message they keep displaying… I’ve been with dreamhost since 2006 and never had a problem like this. Now I can’t even contact support. My brother switched hosts in feb and warned me after an incident, but I stayed because dreamhost has always been good. Waiting patiently as I can.

How long should a restore change like this take?

"
Hello,

Here is an update on the status of the restore for your VPS host machine homie-vserver210. The restore is going well and is progressing without any issues. We apologize for any downtime this may have caused. There is currently some heavy cpu load on the server from the restore process so that might unfortunately be causing sites to load slower than normal. Our admins are looking into that.

As a reminder:

Due to issues with the RAID controller, we have replaced the hardware for this server and we are restoring all data from our backup server to the new hardware. During this process, the data on the backup server will be mounted directly over NFS to allow your hosting services to remain online for the duration of the process.

There may be periods of high load on the server due to this restoration, which is caused by the large amount of data being sent to the server. We will be closely monitoring the server and mitigating that load wherever possible. At this time all VPS guests have been recreated on the new server and the restore is progressing well. All sites are running off mounted backups at this time.

We apologize for the inconvenience caused by this unplanned maintenance. If you have any questions or concerns, please contact Support under Support – Contact Support, in your Web Panel. This notice will also appear on that page in your Web Panel and will be updated throughout the process with the latest details we can provide.

Thank you for your patience!
The Happy DreamHost Server Fixing Team "


#2

Sadly if you read through other threads in the forum you will find out this will probably go on for about a week, some seem to be a little under that, but some seem to be a little over.

Sadly your complaint is common during a hardware change that the message says to contact support if your sites are down, but then support never answers.

This is such a repetitive problem, that I’ve not recommended dreamhost on a couple of projects that I’ve consulted on. If I had a site that depended on daily revenue or SEO rankings, I don’t think I could put it on dreamhost. Sorry Dreamhost, but there has to be a better way you can do this.


#3

THANK YOU LakeRat. Not exactly what I wanted to hear, but good to know. Sending good karma your way.


#4

I seem to have lost email forwarding as of about 4:30 PM EDT, and as of now (11:57 PM EDT) it has not been fixed. My hosted websites seem to be working OK. But now the www.dreamhoststatus.com website seems to be down. What’s going on? This happened back in March and at least once before. My renewal is up in July and if I don’t get a quick response and a fix I am canceling and going elsewhere. This is unacceptable.


#5

Dreamhost status is up, but there is nothing new there recently.

I experienced an email problem and the response I got indicated there where mail delays, supposedly that’s fixed but i’m still waiting on email that hasn’t come yet. How do you know you “lost” it? could it be delayed?


#6

I’m sure it’s just delayed again. The last couple of times it happened the emails started trickling in again after a very long delay of 24 hours or more. Basically I just meant that I have temporarily lost the email forwarding function. But for critical emails, they might as well be lost. I hope they have come through by tomorrow.

Meanwhile I called the tech support line for one and one and talked to an actual human technician who did not mind the fact I did not (yet) have an account, and he assured me that their service should be more reliable. But the most impressive thing was talking to a tech without any delay and no service fees. I was so impressed that I signed up and within an hour I was able to access my new server via sftp, for just $5/month and 90 day money-back guarantee. My Dreamhost account is set to renew in July and when this problem is resolved I will probably cancel. The first year with Dreamhost was “a dream”. The past year, not so much :frowning:


#7

2 days later and all sites are still down. I do have users files and mysql files I backed up from their ‘backup your account’ option, which I did in early June.


#8

What are the URLs of your sites?


#9

I registered in order to respond here.

I’ve been with Dreamhost for a few years now. It’s cheap - about the same sorts of prices of other cheap hosting. But it’s the most unreliable I’ve had out of the lot. It’s been the case that because I’ve not got so much with them it’s a huge hassle to move, so I’ve put it off each time something small has happened. However, I’m now without hosting for 4 days in a row.

I have logged in to find the hardware message:

[quote]Current Outage: Dreamhost VPS host server ‘homie-vserver169’ being moved to new hardware

Hello,
Update: The restore is progressing without any issues. We added an extra processor to the new machine to speed things up a bit. We are also going to start restoring data from the old server, so once the restore is finished, you should see the most recent data before the crash. We will continue serving sites from the backup files through a network connection, so they can remain online, even if somewhat slower than usual, for the duration of the restore. At this time all sites should be up and running. However, if you have issues (other than slower performance) with your site or users, please let us know through Support/Contact Support with the details, so we can assist you.
We appreciate your patience during this process.
As a reminder:
Due to issues with the RAID controller, we have replaced the hardware for this server and we are restoring all data from our backup server to the new hardware. During this process, the data on the backup server will be mounted directly over NFS to allow your hosting services to remain online for the duration of the process.
There may be periods of high load on the server due to this restoration, which is caused by the large amount of data being sent to the server. We will be closely monitoring the server and mitigating that load wherever possible. At this time all VPS guests have been recreated on the new server and the restore is progressing well. All sites are running off mounted backups at this time.
We apologize for the inconvenience caused by this unplanned maintenance. If you have any questions or concerns, please contact Support under Support – Contact Support, in your Web Panel. This notice will also appear on that page in your Web Panel and will be updated throughout the process with the latest details we can provide.
Thank you for your patience!
The Happy DreamHost Server Fixing Team
[/quote]

I have raised a ticket every day, sometimes multiple tickets if I’ve managed to get a response that day. Without fail each ticket has been from some customer support rep telling me that all my sites are fine and are working off backup drives and he can access them just fine. For a period of 12 hours in the middle of the outage my sites were accessible (just slow - but I can deal with that), the rest they were not. For a hosting company to pull off customer support that clearly doesn’t bother checking anything and for them to take so long to replace a machine - shocking. Why they don’t have spare machines ready to go I don’t know - a large hosting company should have these thigns and should expect a certain number of failures. I also don’t feel I’m being particularly harsh, I work in 2nd line technical customer support for a company that offers backbone networks to ISPs.

I’m very lucky though, because I’ve just been commission to purchase and setup a bunch of new VPS’ servers for a new business site. I now know to stay well away from DH. I certainly know our company directors would not accept the responses I’ve had back from them this week.


#10

Hi. I am trying to determine if my site is part of this outage. It went down around 9pm last night. I am getting a “Internal Server Error” (500) when trying to browse my site.

Does this match the symptoms any of you are seeing? Like you guys, I have gotten ZERO response from tech support, even though I pay for the “premium support.” Grrrr.

THey could at least post something on the “system status” page about this.

The strange thing is I have a shared hosting plan. All of my other sites seem to be up and running. That’s why I am trying to determine whether this the issue you guys are discussing, or if it’s something different.

Any info will be greatly appreciated. Thanks!


#11

If you’re on shared hosting, all of your sites are on the same server. If only one of your sites is down, it’s almost certainly a problem with that site, not with the server.

I took a quick look at your domain, cooperti, and it looks as though a security-related WordPress plugin you have installed may have corrupted your .htaccess file. I’ve reverted your .htaccess to the WordPress default (leaving the original file in .htaccess.old), and your site seems to be back in working order.


#12

andrewf! Thanks! I need to “learn to fish” here. I saw some posts related to .htaccess file issues but figure they didn’t apply because I have not applied .htaccess properties to any of my folders. Apparently, Wordpress does(???). Just so I can learn to be self-sufficient here, can you tell me 1) how you found the issue, and 2) how you reset the .htaccess properties to default? Again, I really appreciate your help here.


#13

I have a security-related plugin as well. Could you tell me, or PM me which plugin corrupted that file?

Edit: 6-scan, by any chance?


#14

Yep, that’s the one. This is not a pleasant trend.

If anyone else is affected, here’s a clean copy of the WordPress .htaccess:

[code]# BEGIN WordPress

RewriteEngine On
RewriteBase /
RewriteRule ^index.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]

END WordPress[/code]

Replacing the modified .htaccess with this will let you back into the site. This will probably disable some of the protections added by the 6Scan plugin; I’m not familiar enough with the plugin to say whether they’re necessary, or how to reinstate them.


#15

That is the one. At least I am assuming so because it’s the only security plugin I had been using.

Do you have any clue how to spot and replace corrupted .htaccess files? The CPanel has a tab for .htacces, but only folders/files that show up there are the ones I manually create for protecting client folders. Someone told me the dreamhost WebFTP client shows these … (???)

This was a REALLY bad 12 hour period for me. I’d like to avoid it.

My site blew up when I uploaded a batch of images.


#16

It’s now going on 4 days tomorrow that all sites are down. I received a message from a human that they apologized and would have an admin look at the problem. I’ve heard nothing yet.

I made a ‘system backup’ of users and mysql files in early june from their system backup option. You would think that might help? I’m going to be moving some sites to other hosts as soon as I can. Hopefully my backups work.

ANDREWF… It seems you work at dreamhost. Can you help me out at all?

Moving my sites and splitting them up will take some time. Please work with me dreamhost. I rely on the sites for income.


#17

ok. the sites are back up. Going to split up sites between 3 different hosts.


#18

My site is still down, it’s been over 12 hours. I can’t successfully sftp, even after changing my user (and yes, i’ve waited hours to allow the changes to take effect). The sftp error message is: “unexpected end of string” - neither with my FTP editors or from the web panel using wftp.

The site error is: Unable to load the webpage because the server sent no data.

The site url is: lagymnastics.com

I’ve paid for phone call support but haven’t received a call. No email support either. This is a nightmare.
[hr]
update: 7:10am, 6/30/12

now I’m able to sftp: however, i don’t have permissions to enter the domain folder.

during testing yesterday - since i wasn’t able to modify my user password, i tried creating a new user - and it created the new user on a different server.

so this may be a migration issue. I was getting email over the last 2 weeks saying there’d been hardware failures on ‘zaniah’…

[hr]
update: 7:15 am

zaniah seems to be the issue. I tried adding a user with shell access and received this message:
"OOPS! Something just went terribly wrong: Unable to create shell user: couldn’t find zaniah on this account!!"
Please contact DreamHost Support for details.

the server on which i am able to create a user is: butternut

but that doesn’t help me because the rest of my files are not on that server.

[hr]
update: 7:40 am

i am able to log in via SFTP to my folder.

it’s not a corrupted htaccess file
[hr]
update: 7:50 am

the error at my url (laygmnastics.com) now reads:

[color=#1E90FF]"Internal Server Error:

The server encountered an internal error or misconfiguration and was unable to complete your request.

Please contact the server administrator, webmaster@lagymnastics.com and inform them of the time the error occurred, and anything you might have done that may have caused the error.

More information about this error may be available in the server error log.

Additionally, a 500 Internal Server Error error was encountered while trying to use an ErrorDocument to handle the request."[/color]

=========

The change in being able to successfully (S)FTP and the fact that I’m seeing a different error message suggests that Dreamhost is working to rectify the problem. [NOTE: I never had any problems such as these for over 6 years. Most of that time i was on ‘CENTAURUS’ shared server. Then i experimented with changing to the VPS service recently (and then changing back to shared hosting for financial reasons). That seemed to have some how triggered this cascade of problems over the last 2 weeks.]

[hr]

10:25 am

i’ve received an update from Dreamhost technical support. They are still making repairs. In the meantime, it is suggested that php sites might have a solution. Here is what they are suggesting:

[color=#1E90FF]“If you are having issues with your PHP sites not loading, you might try switching to running PHP as CGI. When running PHP with FastCGI mode there is an additional timeout setting outside the standard PHP settings. You may have better results if you run PHP using CGI mode instead. To switch the php version that you are using you will need to go the Domains > Manage Domains section of the panel. https://panel.dreamhost.com/index.cgi?tree=domain.manage& From there click the Edit wrench under Web Hosting. There you will see “PHP mode” this is where you select which version you want to use. When you are done just press the “Change settings” button.”[/color]


#19

[hr]
1:10 pm

the possible php solution didn’t work. what i’ve done is put up an html page as my index page with a screen shot of my site. the site does render, but takes about 1+ minutes to do so (it’s simply an html file with a 250 kb jpg). so at least my clients are getting to see something…


#20

[hr]

3:30 pm

i received a personalized email from DH tech.

here’s some important tech advice from him:

“Also, it’s best not to change things on the server, like resetting your
password, as this just makes more work for an already overworked machine
that’s restoring.”

[hr]

9:06 am, July 1, 2012

my site is intermittently down this morning. One minute I cannot connect to my phpAdmin; 5 minutes later I can; 2 minutes later I can’t.

i replied to the personalized email. Hopefully they’ll get this sorted out today.

yes, i am working on moving my site to another hosting service. however, i’m running into issues i’ve never dealt with before: 1) how to unzip large compressed file on a server; 2) how to change php.ini size limitations [command line stuff that’s tough for me…]