Scheduled DreamObjects US-West Maintenance - Jan 2nd 10AM (8 hrs)

announcement
dreamobjects

#1

On Tuesday January 2nd from 10AM to 6PM PST, we will be upgrading Ceph, the underlying storage system powering DreamObjects, to the “Luminous” release. The new release includes improved handling of larger buckets and fixes numerous bugs. We expect the upgrade process to last only a few hours but we are allowing for additional time for monitoring and tuning.

During the maintenance, DreamObjects could be temporarily unavailable. However, your data stored in DreamObjects will remain safe.

We will closely monitor the maintenance and will be posting periodic updates on the DreamHost Status Blog. This maintenance only affects DreamObjects US-West.

If you have any questions or concerns, please don’t hesitate to contact support for assistance.


#2

Does that mean the API won’t be accessible, or object URLs will not function?


#3

It’s possible for both the API and the object URLs to be temporarily unavailable. During past upgrades, there is a small blip when services are restarted with the new software and in occasionally when the cluster needs to rebalance data after the upgrade.

Testing the Luminous upgrade in our test and staging clusters has gone smoothly with minimal disruption. We’ll work to ensure the same happens with the production DreamObjects cluster too.


#4

From about 1:10 PM the object URL and API have been unavailable.
-mort


#5

I was a little surprised by the potential for downtime, and now by the actual downtime.

Is DreamObjects not meant to be used, in part, for static website content as well as object storage like uploaded user data via a website? If one used DreamObjects in such a way, during this maintenance window their site would be broken.

Maybe I’m missing the appropriate use of DreamObjects, as I know CDNs are often put in front of things like this, but even in that case new content couldn’t be uploaded.

Luckily I don’t use DreamObjects for anything that I need available all the time, but just the fact that I’m saying this I feel is damning for the usefulness of the service. Am I wrong here?


#6

Hi @mort, we’re really sorry for the trouble. Unfortunately the team encountered unexpected issues during the upgrade. DreamHost Status has been updated. Please get the latest updates on the the maintenance page

https://www.dreamhoststatus.com/pages/maintenance/575f0f606826303142000510/5a395f11994f1104d9832a0b


#7

Yes, thanks, I was watching.
Today there seem to be new problems. When I upload images they take a long time or time-out so that in a recent test it took 45 minutes to upload 50 images, but only 12 of the images show up at the expected url.
Also I have an application that uses a perl module
Amazon::S3::FastUploader
which lets you upload a bunch of files in parallel. That has been working fine until yesterday’s maintenance. Today it never completes. Something like this happened in September; also after maintenance, and you guys seemed to get it fixed.
If instead of the FastUploader, I run a loop using add_key_filename, It seems to work, but takes 10 minutes to upload 10 images.
I’m submitting a ticket.
-mort


#8

This is one of the most common use case, indeed. The downtime should have been very minimal but unfortunately the team hit an unexpected issue.

The team is working on implementing the changes needed to avoid any downtime in future upgrades.


#9

Thanks for this - I was worried my message came off sounding more critical than intended. I’m happy to support open source-based efforts like DreamObjects, and fingers crossed this provided lots of learning opportunities!


#10

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.