DreamObjects are down

dreamobjects

#1

That’s all I know. Since about 4 AM my website which serves content from DreamObjects has been throwing time out errors in the logs, and a few pages show broken images. This will kill my business.


#2

Hi mort, our system engineering team has been investigating intermittent connectivity issues this morning. I apologize for the impact this has had on you and your site. I’ll be in touch with you soon.


#3

The IPv4 networking issues affecting DreamObjects have been resolved. There is an outstanding issue with IPv6 the team is still working on.


#4

OK, thanks for the news. That outage lasted from 4 AM to 6 AM which didn’t affect me much.
But there is another intermittent connection problem now. Is this just me?
This time it only lasted as long as it took to write this post - maybe 10 minutes.


#5

My apologies, I missed your message. Looks like we did have a small issue yesterday with our SSL termination. As you mentioned, it was short. We’re actively working on both better monitoring as well as investigating better SSL termination alternatives to replace it.


#6

June 13, 9:00 AM or a little after. DreamObjects is throwing 504 gateway error messages.
-mort


#7

There is an ongoing issue with DreamObjects today. You can follow the status here http://www.dreamhoststatus.com

The team is working very hard to resolve the problem.


#8

And it got better about 4 PM. But 10:45 PM, it’s down again:
https://objects.dreamhost.com/argentum/ARU/ogac/chm47/previews/AST_7431.jpg
shows this:
503 Service Unavailable
No server is available to handle this request.
-mort


#9

3:20 AM 503 errors again.
It was intermittent from about 10 PM, better around midnight…
[hr]
4:15 still 503


#10

Yes, the cluster is in a much healthier state, but not all tests are passing with 100% success. The team is still working on making it fully performant again.


#11

Wow. 25 hours:
“Update June 14th, 2014, 10:20 AM PDT: Stability on the DreamObjects cluster has been restored. Requests appear to be resolving properly now that the system has had time to re-balance itself. Our test are reporting properly now. If you do have any questions or concerns, please contact support…”

Everything works fine for me.


#12

Yes, it was a painful outage. We’re working on a full post-mortem and will be sharing it. Look for that early next week.


#13

Here’s the post-mortem on the outage we experienced over the 13th and 14th http://www.dreamhost.com/dreamscape/2014/06/24/update-on-dreamobjects-service-disruption/


#14

This was at about noon.
Here is a new wrinkle. I get this error trying to view dream objects using https
"An error occurred during a connection to objects.dreamhost.com. Peer’s Certificate has been revoked. (Error code: sec_error_revoked_certificate)"

So this doesn’t work:
https://objects.dreamhost.com/argentum/ARU/ohab/van21/thumbnails/_MG_2095.jpg
but this does:

Hope this is a quickie.

Weirder. update:
For me the https works in Chrome and Safari but not in Firefox or IE. (Windows)
It could be how my browsers accept ssl certs or not.

And it seems to be fixed now, before 1:30PM

-mort


#15

Mort, thanks for the report, it was a brief blip due to a mismatched SSL cert. We’ve fixed it.