DreamCompute - Disk Space Quota Bottleneck - Confusing and Very Expensive


Because Dreamhost is disabling my sudo access to VPS, I’m being forced into DreamCompute and I have some serious concerns with high costs and disk space.

I currently use 366GB of disk space and 777GB of bandwidth each month on my VPS plan (unlimited disk space and bandwidth - features that I’ve had for 10 years, even back in the internet “stone age”). All of my sites run fine with 800MB of RAM and I’m assuming just one CPU…

With this new DreamCompute plan, it says I have to pay $10/mo for 1GB RAM, 1 vCPU and 80GB of disk space (explains absolutely nothing about bandwidth on the sales page or what you need to do when you need more disk space). Since I have 366GB being used (mostly product images), I would be forced into a $50/mo plan with 500GB of space, which at the bare minimum would be able to handle my 366GB of disk space. I wouldn’t need anywhere near the 8GB of RAM and 4 CPUs that same plan offers.

I looked into DreamObjects which charges $0.025/GB + $0.05/GB outbound bandwidth. With my current usage, this comes out being nearly $60/mo just for raw disk space and bandwidth… which is unacceptable. I don’t even know if DreamObjects is even supposed to be used as a supplement with DreamCompute when there isn’t enough block storage disk space available. I’m also told that DreamObjects is a completely separate service - very confusing and expensive.

Question: If I sign up for the $10/mo DreamCompute plan, isn’t there a slider where I can simply increase my disk space with a reasonable cost? and WITHOUT having to sign up for DreamObjects? I’m not really getting a clear answer from Dreamhost support either.

Kind regards

Hi Peppy

let’s start by saying that DreamCompute (and DreamObjects) and VPS are two quite different types of products, they come from two different type of philosophies: they seem to do the same thing but their architectures (and as a consequence, their costs) are different. If you really can’t do without sudo (note: I’d be glad to discuss more with you to find out if there are other ways to achieve your goals without sudo) then you may achieve the same results of your current VPS usage by changing a little how you do things and DreamCompute + DreamObjects.

The advantage of DreamObjects is that it provides automatic replication and availability that no single disk (virtual or real) can provide. The price you mention is meant for sporadic usage but for the kind of usage you have, a pre-paid plan would work better. For example, the 1,024 GB plan is only $10.95 per month and it saves you from having to deal with data replication and availability, for example. On the bandwidth side, remember that upload is not charged on DreamObjects: is your 777Gb bandwidth all in download? In the worst case scenario that’s about $40 per month of download bandwidth.

Freeing disk space from the file system would allow you to save money on the compute side, where you say a small instance would be enough and cost under the current plans $10/month. that would be a total of around $10+40+10, roughly $60 a month, if I’m not mistaken.

What kind of application do you run on your systems now?

Greetings Smaffulli,

Thanks for the response. We mostly run two large marketplace sites. Here is our largest one: http://goo.gl/ju2PH . This is the site where members add products for sale and images of their products, which makes up the bulk of disk space. Unfortunately, we barely get any traffic form this site anymore due to search traffic issues… Our other marketplace is smaller, but it makes up the bulk of the bandwidth as it gets more visitors. Most of the bandwidth usage probably comes from images loading up or search engine indexers loading pages.

The problem with “managed” VPS for me is the way Dreamhost deals with the FcgidMaxProcessesPerClass variable in my httpd.conf file. It gradually spawns more and more processes until the RAM gets maxed out and the server crashes. This happened every 30 minutes and it didn’t matter if I had my RAM set at 1GB or 4GB, it gets used up or allocated into “cached” memory and it just crashes…

This is why sudo access is required for me, so I can place a hard limit cap on the “FcgidMaxProcessesPerClass” variable and prevent the server from crashing. My sites gradually grow, so I simply use the slider to increase my RAM every so often to account for site growth if crash issues start happening. Right now it’s fine tuned. The sacrifice: sometimes the site will get slightly slower with limited processes running when we get heavier than normal traffic, but it prevents the server from maxing out memory and crashing.

EDIT: I can’t remember if I used sudo or not for this, but I may have had to use sudo to install sphinx search into my sites. This feature was absolutely imperative for my sites. Again, it’s been awhile so I don’t know if sudo was required for this or not.

I see two paths for you to go forward:

  1. investigate how to optimize your applications, maybe split the ones that are more ‘noisy’ to a separate server
  2. investigate further modernization of the infrastructure, using a mix of VPS, DreamCompute and DreamObjects.

If I understand correctly, you’re hosting all your sites on a single VPS instance, is that correct? If that’s the case, maybe there are optimizations you can do on the application side, leaving the server’s configuration as is. Have you profiled the applications to see where they’re likely to get stuck? You could try installing on your current VPS something like New Relic to get more details about your applications and databases. You may discover important things that will make your problems go away, without having to fiddle with apache conf and maybe even save some bucks on RAM you don’t need.

For option 2, I think the first step would be to understand exactly how your bandiwidth is split between upload and download. Maybe most of you traffic is made by your users uploading their images to the stores and those are not charged by DreamObjects. Maybe it makes sense for at least some of your sites to start using DreamObjects instead of your local disk storage, so you don’t have to worry about keeping multiple copies of your data.


Thanks for the reply. I am hosting around 60 sites on VPS, although all of them but 2 are tiny and don’t get much traffic.

Unfortunately, I have very little experience in profiling applications or doing any backend apache stuff. I don’t know how to manage any of the server-end stuff. I had to teach myself how to use sudo just to fiddle with the apache conf file to fix the server crash issues. That “New Relic” software looks like it costs $150/mo, which is too much for me. I’ve been trying to find a good open source product, some of these might also require “sudo” access which won’t be available on VPS soon… Although, it would be nice to learn how to find some of this server performance information and learn more server-end stuff.

Currently, I use the “top -c” shell command to view the running processes for each site and memory usage. When I write a script, I’ll run it and watch it here. I think the problem Dreamhost has is it sets a high limit on the number of processes that run (the php54.cgi processes). So I had to set a hard limit of 6 processes.

What happens: visitors view the site, processes start up and they don’t last long and usually don’t use too much memory, then they sit around in “sleep” mode until the expiration time is up. During normal traffic levels, I assume Dreamhost is trying to make the site load as FAST AS POSSIBLE for all visitors, so it spawns 8, 10, 12+ processes, with many of them ending up in “sleep” mode. Each php54.cgi process is usually around 20 MB, but it can go up to 80MB each when processing scripts for loading up pages where images need to be resized for thumbnails. Too many processes are spawned, even while some processes are still “sleeping”, then the memory gets maxed out and server crashes…

I found that limiting the processes to 6 (80MB * 6) prevents a memory crash and these 6 processes allows the site to load up with reasonable speed. The drawback is it slows the site down (slightly) during heavier traffic loads when all 6 processes are running hard trying to handle all traffic… I found that 6 processes running frequently with reasonable loading time, is better than many processes running “not so often” and using up lots of memory (basically a site that loads up super quick for everyone, but crashes every 30 minutes).

I did some investigation into my disk space and was able to delete about 100GB off my account, which consisted of directories of daily MySQL backups, temporary image cache directories that built up, and large files I used on previous projects but don’t use anymore. It turns out I only had 120GB of actual images + 88GB of cached rendered resized images (208 GB total) for both major sites combined, which is much less than I expected.

Unfortunately, I have no idea how to find inbound/outbound bandwidth. Members upload images regularly, but I doubt it’s very much. I’m guessing that probably 95% of the bandwidth is outbound from visitors and search engines browsing the site (images loading up on page views). The biggest and most expensive problem with DreamCompute/DreamStorage appears to be bandwidth costs.

I think if bandwidth was free/unlimited like it was in 2006 (internet stone age), moving to DreamCompute + DreamObjects might end up being an excellent deal. You would think that with advances in technology and infrastructure, bandwidth costs would go down… as would disk space and RAM… $48/month in bandwidth charges alone is ridiculous. That’s more than both my current monthly VPS and MySQL VPS costs combined and then some.