DreamObjects stalling out

dreamobjects

#1

Cloud storage is having unacceptable delays today. Since about 6:30 AM the time to upload 10 files has gone from about 1 second to over 300 seconds, generally around 100 seconds. I’ve complained to support.
This may be the first slowdown for the east cluster. It’s probably a bad sign.


-mort


#2

Hi Mort,

Thank you for contacting us. For help with DreamObjects please contact our support team via the panel here https://panel.dreamhost.com/index.cgi?tree=support.msg& they will be able to get the issue over to one of our cloud specialist to assist further,

Thanks!
Matt C


#3

As I said, I did complain to support. They got it fixed…

Due to some
hardware issues, Ceph (the platform that DreamObjects is built on) had to
shuffle data around to accommodate recently replaced hardware.
Unfortunately, DreamObjects is designed to ensure that data integrity is
the highest priority, so requests for content during data shuffling
periods like this can occasionally take longer to load or even result in
timeouts.

Fortunately, as of a few hours ago, that process has completed
successfully and you should see a dramatic reduction in the load/response
time for objects in your bucket(s) now! However if you continue to have
issues with DreamObjects we would greatly appreciate you notifying us as
that is likely a separate/unique issue that needs further investigation.

But then yesterday I saw something similar happening, and after I complained I saw that the status page had a notice that they were working on it, from about 12 hours before I complained. So The status page was way ahead of me.
But this problem is ongoing.
-mort


#4

February 20, 2019 2:22PM PST
[Resolved] This incident has been resolved, and systems are fully operational. Please contact support via the DreamHost panel if you are still experiencing related issues.

OK!


#5

Something is happening.


-mort


#6

OK, that got better for a couple days. Today some slow down or drop outs, some customer complaints. But the graph is something else. There is a good window to upload or download just about every 40 minutes. What does that mean?


-mort


#7

Still problems. Is that 40 minute cycle my diagnostic or is it real?
This is a different measure, time to get a list of 50 recent objects, so far today:


-mort


#8

Down again.
At first it got a lot better. Just about normal for most of a day. But for the last 3 hours or so it’s been slowing down and at about noon it stopped responding to list in objects or uploading or downloading.
No mention on the status page.
-mort


#9

February 27, 2019 1:48PM PST
[Investigating] Our cloud engineers are investigating an issue resulting in a service disruption for DreamObjects customers. We are closely monitoring the situation and will be posting periodic updates as further information becomes available.


#10

Then at 2:00PM it got way better!
Yay!
-mort


#11

Yesterday & today very spotty:


fudge.
-mort


#12

Hello @mort,

Just wanted to throw my voice in here, having issues with DreamObjects as well. I have a support ticket in.

Waiting on a manual upload to my bucket now using s3cmd…stuck on a waiting x+ sec… loop (handshake operation timed out) is the only error I’m getting. :frowning:

Update - After 3 operation timed out errors, it uploaded my files.


#14

Well add me too in the list of people facing this problem.I mean i m not facing this much variations but still lot more than before…I hope it will be solved soon.