DreamObjects Down

dreamobjects

#1

Is anyone else’s DreamObjects down? The system status page says on the 26th a problem was resolved, but everything is down for me on the 29th.

I am really hoping this does not turn into a thing.


#2

DreamObjects did have an issue overnight. Here is the status post from the event http://www.dreamhoststatus.com/2014/08/29/dreamobjects-connectivity-issues-2/

There was an issue where one OSD was reporting as 927 million percent full and so Ceph thought the entire cluster was full. The team was able to resolve the issue and ensure data was redistributed from the problematic OSD.


#3

Seems to be working now.


#4

Since day 29, 13:00 UTC I’m getting slower times accessing dreamobjects, anyone else with the same problem?


#5

Can you create a support ticket with details (if you haven’t done so already)? Specifically I’d like to know if you’re connecting over IPv6 and if you’re connecting from one of our hosting machines or elsewhere.

Thanks!

Justin


#6

I’m testing from DigitalOcean (NYC) and using IPv4. Some times in ms retrieving random objects (small objects):

325
318
5500
5315
5404
5329
495
10624
5854
10505
5416
5468
550
5412
5439
402
5584
10398
5330

The 300-500ms are normal.

Edit: testing from my home the times are all normal, I’ll query DigitalOcean to check if it is a problem on their side.


#7

I only use DreamObjects for scheduled backups (from DigitalOcean in SanFran), so I do not really pay much attention to speed; however, after DreamObjects were down the other day I had to go through and do some manual backups, and it really seemed to take forever…maybe 10 minutes for each one, where if I did the same backup on a local disk it would have taken one minute.

I use rsync, so I guessing the slowness comes from rsync checking all of the files for diff.

But yes, very slow. But for scheduled backups, I am not very concerned about it.


#8

How are you using rsync with DreamObjects? If you’re using it via s3fs (or something similar), that will be rather slow, yes…

You may want to consider using something like boto_rsync, or consider using a utility that’s specific to handling backups, such as duplicity. (duplicity has the added benefit of allowing you to restore old snapshots, so there’s that too.)


#9

Yes, s3fs. But it works fine for my scheduled backups. The only reason I noticed it is when DreamObjects were down and I had to run some manual backups after the problem was fixed.


#10

Solved. The problem was a slow DNS server (4.2.2.2).