Is there a recommended max number of objects per bucket?

dreamobjects

#1

Is there a recommended max number of objects per bucket?

I’m building a web-based system that points to publicly viewable JPG images stored on DreamObjects. Each JPG is loose in the bucket. There are 8,000 JPGs in that bucket now… and soon, there could be as many as 200,000.

Should I build my system differently? Such as: subfolders each holding a subset of JPGs?


#2

I’m not aware of any recommended practices for this. Compared to a real file system, where it does make sense to have subfolders of subfolders to ease in browsing, listing files and preventing reaching files-per-directory limits, DreamObjects is different.

Every file you upload, doesn’t go into a directory but the name of the file has the full path in it instead. So one could imagine it as 200,000 files in the / directory, and they just happen to have some paths in their file names, so when you browse a specific directory in a S3 client, it searches all files for that path in their name to display them like they are in a directory.

So in general, do what is easier for you for organizing your data and you should be fine. :slight_smile:


#3

When you start getting tons of objects in a bucket, listing the objects in the bucket starts to get increasingly difficult. What I would recommend would be to “shard” the objects across several buckets.

You can easily continue storing hundreds of thousands (even millions) of objects in a single bucket if you like of course. But sharding them up across a bunch of buckets gives you some more headroom as you continue to scale up.


#4

Thank you DreamHost team! Your responses are very helpful.


#5

My 2c. I have 10+ million objects in a single bucket and no problem at all, except listing, of course! :slight_smile: