Directory Folder and Image Limits - ext3 or ext4?


#1

Hello,

I have a site where members can upload photos of their items. Lately, the site has been getting larger with thousands of members and tens of thousands of images. All of the images are stored in one large directory folder I am getting concerned about any kind of limits that may be involved.

Does Dreamhost have an “ext3” or “ext4” system? What are the current limits to the number of images that can be stored in a folder or directory? In addition, what is the limit to the number of folders/directories that can be stored within a directory?

Any advice for setting up a directory system for organizing images for millions of potential members in the future (scalability)?

Thanks
Kind regards


#2

If you’re expecting to deal with a potentially large number of uploaded files, you’d be best off storing those in an external service, such as DreamObjects. While our servers are all set up to use the XFS file system, which can handle large numbers of files in a single directory, performance will still be poor on large directories (especially when performing operations which involve all of them, like viewing the directory!), and having a very large directory like this will impair some functions relating to your account, such as creating backups.


#3

Greetings,

This original post is 2+ years old - does Dreamhost VPS still use XFS filesystem? I am beginning planning on a new project which involves a lot of image uploading. My new plan is to spread the images across a number of directories/sub-directories. Based on experience, what would be an optimal number of images to store in a single directory and not see a drop in performance or impair maintenance functions (based on the current Dreamhost VPS file system being used)? I was thinking maybe 100,000 images per folder (each image being around 25kb in size), although maybe it can handle a bit more or a bit less?

Thanks
Kind regards


#4

I would strongly suggest you to look at DreamObjects with more attention, given your use case and the fact that you’re starting a new project. DreamObjects it’s designed for serving web content, exactly for your purpose to efficiently store/retrieve lots of small objects (images, in your case) via HTTP.

You won’t have to deal with all the nasty stuff that comes with a file system (redundancy, size limits, efficient distribution, etc): just drop files in the object store and let the DreamHost backend replicate your files (redudancy). You’ll never have to care about disk sizes or disk speed… and you can also put all your images in a CDN with the flip of a switch, no sweat.


#5

I agree with the above post, DreamObjects will be a good solution for this use case.

DreamHost VPS machines still use XFS, even those that are converted to SSD storage, which is a majority of them. Even so, the disk space included with the VPS is no longer unlimited for new VPS SDD setups, and so depending on what the storage adds up to, it may require increasing the cost and memory and disk to store more data. With DreamObjects there is less worry about how many files can go in a folder, as cloud storage isn’t a real file system and is designed to store larger amounts of files.

If you have further questions about how to distribute your files, or how many total files you expect, on DreamObjects, we can help with storage recommendations.