Uploading blocked

dreamobjects

#1

Got this disturbing message today:

[quote]I am reaching out to let you know that my cloud developers have informed
me that they have temporarily restricted access to your DreamObjects
buckets “2015argentum and argentum”.

The reason is because there is 2.7 TB of data and a total of 33 million
items which is degrading performance on our indexing service and causing
performance issues with your buckets on the servers themselves.

Once they resolve the issue, your buckets will be restored. If you have
any questions, please feel free to reach back out to us.

Thanks,

Joseph R.[/quote]

And we can’t upload to our current bucket or even a new bucket. I’ve contacted support, but who knows when they will get back to me or help. I told them that 33 million items doesn’t seem like a lot to me. I remember this from 2 years ago, In February of 2015 there was a post to this forum titled "Is there a recommended max number of objects per bucket?"
JustinK said then there wasn’t a limit, but I guess he was thinking about hundreds of thousands, not millions. cleverdevil offered the advice about sharding over many different buckets, which I have a plan for, but it does seem to require a data base to remember where any object gets put if you have to put a days worth of uploads in a thousand different buckets.

I did notice degraded performance the last few days, I guess it my own damn fault. I’m sorry about that. I’ll work on my sharding scheme tonight.
-mort


#2

Hi Mort, sorry for the issue here. 33M objects in a bucket should not be a huge issue in modern buckets, created with a modern version of Ceph … but I suspect your bucket was created with an early-days version of Ceph. At the time, there were no buckets with millions of objects and some issues were unknown … now Ceph handles large sizes better.

Your bucket has been updated now, you should have received an email from support. We still recommend people not go over 1M items in a bucket in general because listing bucket of this size will always be problematic. Sharding and storing an external index of your buckets, that you can search fast, will help most users.


#3

Sincerely, Thank You.

OK!
-mort