I got this from support today:
I wanted to update you and let you know that we have removed any quotas
from your account. It is important though that you work to re-organize
the data and prevent putting 10’s of millions of objects into a single
bucket. Due to limitations with the Ceph system the metadata can grow so
large and be in a single file (not sharded) that it can cause disruptions
in service to DreamObjects, and so we have to suggest to customers, and
in extreme situations put on quotas to avoid disruption of service. If
you can shard your data, distributing it between multiple buckets, it
would help to prevent stability issues with DreamObjects as well as make
your access faster.
We are awaiting later versions of Ceph that will have sharding for
metadata built in so that it can handle larger number of objects.
But last February in a post titled
Is there a recommended max number of objects per bucket?
one reply said
[quote]I’m not aware of any recommended practices for this. Compared to a real file system, where it does make sense to have subfolders of subfolders to ease in browsing, listing files and preventing reaching files-per-directory limits, DreamObjects is different.
Every file you upload, doesn’t go into a directory but the name of the file has the full path in it instead. So one could imagine it as 200,000 files in the / directory, and they just happen to have some paths in their file names, so when you browse a specific directory in a S3 client, it searches all files for that path in their name to display them like they are in a directory.
So in general, do what is easier for you for organizing your data and you should be fine.[/quote]