Funny how you describe "normal"


#1

Nearly 2 hours ago, everything went down across all my sites. I saw on status.dreamhost.com that they were having problems and the message wasn’t updated until a few minutes ago (nearly 2 hours later) with this message:

“Network Problems Resolved
The network problems described earlier have subsided. A switch error appears to have been the cause. We’re still investigating, but again, things are normal and we’re keeping a close eye!”

Email is getting through somewhat BUT EVERY SINGLE WEBSITE IN MY DOMAIN IS STILL UNREACHABLE. That is not NORMAL to me by any stretch. Yeah, I just submitted a support ticket since panel.dreamhost.com seems to be back up…I wonder if I’ll find a new host before I actually get an answer. :wink:


#2

Yeah, seems dreamhost.com itself is back up, but none of my domains are either.


#3

Not only are all my websites offline, but all the ones by everyone I know who has a DreamHost website… it’s a little embarrassing after having convinced them to go with DreamHost in the first place, but I’ve never used a web host that didn’t have outages from time to time.


#4

I’ve stopped suggesting DH to friends…I continue to use it myself, but my patience is wearing thinner by the outage.


#5

Yes, same here - except that I can reach the webmail interface, but not my sites and I can’t get FTP either. That’s not my definition of “normal”, and it’s very frustrating.


#6

Over the last 10 minutes or so, all of my sites have “started” to come back to life. Some still can’t connect to their respective MySQL database, but at least they are resolving. I suspect within the next 15-30 minutes or so, everything will be back… sigh…
rlparker


#7

Whatever was fixed must be propagating throughout the servers. My sites were up but the ftp was not functioning. Now ftp and http are both working.


#8

now “that” describe a “normal” re-birthing process…loading…stuff will come back on line incrementally and over a bit of time…for other’s reading, depending upon your sites’ construction, make sure you clear you caches (possible even DNS, depending on how you are set up) in order to see “actual” status (especiall using Internet Exploded or any form of AOL)

rlparker


#9

I’m back up too. Still not happy and DH owes us a better explanation. A two-hour network-wide outtage is not something to take likely, especially when it happens during business hours. I can almost forgive a power outtage that was out of their control, but a “switch error”. Come on!!!


#10

I hear you…and you have a point…as for me and my clients, I’m hanging in there (though I have been “hanging out there” a bit more than I would like lately) in hope DH gets a handle on this stuff. They have been exceptional in the past (I’ve been with them since 1997!), and I am hopeful/confident? they will get back to that level soon (please, pretty please, with sugar on top?). They have had “bumbs” before, and always managed to pull it tigether, so I am hopeful. Lately, though, it has been discouraging…especially email.

rlparker


#11

well I don’t know if this is related to the outages this morning, but two of my sites got Brazillian Boy hacked about an hour ago. I hope that DH has some recent backups to restore from…


#12

backups are made on an hourly basis. Just log into FTP and then move into the hidden directory called .snapshot You’ll find Backups!

Mysql also have backups regurarly, but you’ll have to contact support if you need access to that.

-Matttail


#13

They should, as I have never found their backups to be lacking. Depending on whether or not your sites are database driven you may need to reload you MySQL databases.

Just a friendly note:

I strongly suggest that, in spite of DH good record with back-ups, you develop and implement your own backup plan, because “sh*t does happen”. I would never rely exclusively on a web hosting providers back-up for any critical site. That is our responsiblity as webmasters.

I would also be curious to know what was hacked, and how. Are you using PHPbb, AwStats, etc., and is all your code current/patched?

Regards,
rlparker


#14

I’m really getting so sick of all the problems. And my site is still not back up! I’ve tried FTP and www, both still down. I have to say not only have I stopped recommending dreamhost to friends, I’m seriously considering moving myself. I came here because my friends raved about dreamhost - I can’t say I agree with my experience so far.

*EDIT - site finally came back up just now.


#15

Your post was probably more useful to the poster than mine; you told when how to find the back-ups which I just “brain farted” and forgot to include…thanks!

I still think, though, that none of us should rely on those backups…if there is prolonged lack of connectivity, we won’t be able to get to the backups, and if we have to get the site back up ASAP on a different host, we are “out of luck” if the only backups we have are on DH, which can’t be reached.

–rlparker


#16

That is an understandable position to take, especially if you are “new” to Dreamhost, as things have been worse of late and your experience would be heavily negative in the “short term”. For me, while recent events have been problematic, to say the least, over the 5 years I have hosted numerous sites with DH the total downtime has been insignificant.

Now, it is never “insignificant” if you are “down” when you need to be “up”, but these are machines, and they burp, hiccup, belch, fart and break…no matter who’s machines they are…

I think it also behooves us all to remember something that we all too often forget (primarily, I believe, because DH works so well most of the time):

No *mission critical" site should ever be hosted on “shared servers”.

In a very real sense, we all “get what we pay for” and while I sympathize with anyone whose site is broken, if 100% uptime is critical to you (or 99.999% or 97% or whatever your "pain threshold allows) the proper business decision is to suck it up and spend some real money to arrange your own connectivity, redundancy, storage, mail, etc…

For me, all this equates to DH still being a reasonable compromise between expense and functionality. The low cost allows some sites to operate profitably that would be completely in the red if they had to provide their own infrastructure.

Now, having said all that, I think it is important that DH service does seem to be deteriorated of late, and if it continues, at some point, the cost/functionality ratio will change. It is my exsperience that DH, in the past, has managed to pull it together before I, or my clients, have felt it was time to change.

As usual, this is just my opinion, and YMMV…

-rlparker


#17

Thanks for the pointer to the backups - I’d no idea they were there. I do have local backups, not recent but I hadn’t made any changes recently so they probably would have sufficed.

They didn’t get to the database, thankfully, but they did delete a slew of files in my web root directory as well as some image folders, etc. Seemed like random vandalism. I’m almost fully restored now. I’ve no idea how they got in, though. I’m running the Nucleus CMS but besides that, nothing that wasn’t installed by DH in the basic package.

Bastards.


#18

Thanks for the update, and I’m glad you are gonna recover! I just happened to be on the Nucleous website last night, and noticed a series of security updates recently added. I don’t know how recently you have checked over there, but it might be worth a visit to see that you are running the latest code “patched” for the most recently discovered exploits.

Cheers,
rlparker


#19

“No *mission critical” site should ever be hosted on “shared servers”."

I agree with this, however … that isn’t the problem lately (or hardly ever, in our experience over the last 2 years).

Their network keeps failing. Big time. Wouldn’t matter if you were paying them $100 a month for their dedicated server option - you’d still be hosed. This is twice in a two month period that they’ve had a serious outage related to the network. Luckily this one didn’t last an entire day like the last one. They also had another one several months ago (That’s 3 in a one year period).

And nevermind the ongoing email problems which never seem to get fixed (for example, just now I didn’t get an email that my business partner did - and it was sent to an email alias that goes to the both of us)

That’s the reason we’re looking at other options. I love the features you get at DH, their support people seem very good and usually reply quick to trouble tickets, and of course the price is great … but with a flakey network, none of that really matters :frowning:

Everyone is going to have problems … and once a year would be acceptable (IMHO), but when things keep breaking (especially with increasing frequency), you start to lose faith.

  • Brian Roach

#20

Broach,

All your points are valid, but I think I was not clear in my statement that you quoted.

“No *mission critical” site should ever be hosted on “shared servers”.

was only stated as a general comment, and is true, as far as it goes. The real “meat” of what I was trying to say was:

“the proper business decision is to suck it up and spend some real money to arrange your own connectivity, redundancy, storage, mail, etc…”

I just assumed (always a dumb thing to do!) that my meaning was taking responsibility for your own infrastructure - not just 'upping" the ante on a commercial hosting service.

The real money I was talking about is several orders of magnitude above a $100/month “dedicated server”, and I see now that I wasn’t clear.

What I was referring to was leasing your own pipes, arranging for necessary connectivity and peering arrangement and back-up, fault tolerant archetecture for you redundant web, mail, and DNS servers, ad nauseum…in short, the whole magilla.

Big, Big, bucks…

While a dedicated server at DH or another hosting service may be more robust than a shared server, it alone is not acceptable for a mission critical site precisely because it is subject, as you pointed out, to many of the same frailties as a “hosted” shared server: it all depends on the host.

Hosting yourself (if you can afford it, have the technical expertise to design it and maintain it, and are willing to accept the full resposibility when you missed something or when something breaks) is how those kind of sites should be run. Everything else involves some kind of cost/performance tradeoff, and only you can decide what is a resonable compromise between cost and fuctionality/redundancy.

–rlparker