Beware of "unlimited" storage


#1

I’ve been a happy customer of DreamHost since May 2006. That’s quite a long time! :slight_smile: I’ve endured server failures and dreadful nightmares as 100+ websites hosted here suffered all kinds of catastrophes. I endured service upgrades that rendered sites unavailable until I managed to fix them to run again. Nevertheless, and in spite of all those troubles, I persisted as a faithful customer. I’ve blogged about DreamHost and the reasons I stay with them quite often (one of the many reasons is the technical support, who more often sends full reports with suggested performance tweaks instead of simple, automated answers). I’ve recommended it to several customers. I’ve compared performance and services with many other providers, and while sometimes they beat DH on some things, it’s hard to beat them on price and performance simultaneously — and hardly anyone else has such a great backoffice with so many features, nor the kind of support that DH has.

Years ago, I started beta testing several bleeding edge features for DH (the last of which was DreamObjects). As a consequence, I was among the first hundreds to benefit from “unlimited” access — unlimited users, unlimited domains, unlimited websites, unlimited disk space, unlimited traffic. At the time, such services were quite unusual and very innovative.

This encouraged me to add even more websites to my account. I host mostly non-profit organisations with very limited budgets and a handful of personal websites for me and a few friends. Occasionally I also develop “demo” sites for potential customers. And often, even if the customer goes away, I keep them around anyway, since there might have been some custom programming that I might use elsewhere. My WordPress plugins, some of which sponsored by customers, were all first tried and tested on DH.

I was also a user of DH’s streaming server solution. Aye, I know it wasn’t a big deal. But it was quite useful while it existed. It also allowed me to do things for potential customers and non-profits that weren’t possible on “other” professional video streaming solutions, namely, the ability to select videos for specific registered users. Of course you can use high-end video services for that, at insane costs. DH was so much cheaper.

A year ago or so, I built a WP-based training site, which offers educational videos for a selection of registered users for a tiny non-profit. There are really just a handful of students. But the non-profit required a very simple way to upload the videos (you cannot use WP’s Media upload feature for that, since it’s limited to a handful of MBytes), as well as a way to restrict certain users to view just some sets of videos/audio/files but not others. This required a rather complex setup, but I managed to get it working. It had little traffic — as said, this was just for a non-profit with a handful of users — but, because videos (and other multimedia files) are huge, this meant they would take up a lot of disk space.

In more recent times, DH has updated their terms of service and acceptable use policies, without my noticing. I know that’s my fault, not DH’s. Now it’s strictly forbidden to make backups. But flagging something as “backup” instead of “content” is at the discretion of DreamHost’s technicians.

The support team is also unavailable to discuss their interpretation of what is a backup and what is not. I mean, a content-less website running from WordPress is probably not a backup. But as soon you store some multimedia files there — images, for example, to be shown with the content — those media files might be considered “backup” and that website deleted.

I thought that the main issue was just that the videos took hundreds of GBytes of hard disk space, and their servers might be running low on that. Ok, I can accept that argument. So I moved them to the competition — these days, for the monthly cost of what DH asks for shared hosting, you can get your own unmanaged hardware (no, I’m not talking about VPS — these always have restricted disk space and metered traffic. I’m talking about real, physical hardware connected to an unmetered Ethernet port. Aye, it’s that cheap these days. Much cheaper than DreamObjects, in fact. Of course you don’t get any management and no nice control panels. But you can install them).

But DH did not stop at that. They started looking at all my websites, and yes, you guessed it, they flagged one of the WordPress wp-content directories on another website for a non-profit as “backups”. In this case, there were no videos, just a lot of images. Quite a lot of them. So, once again, this meant pushing yet another website away from DreamHost.

I suddenly realised that DreamHost can do that for all my sites, if they wish. You see, I have really survived many nightmares with DH. Most of them were fixed after a few days — once it took a week, sometimes a month, others were never fixed — and I rely on some WordPress tricks to be able to successfully restore those websites and continue to offer the service. Now it appears that I’m not allowed to do so: all that is now seen as “backups” and, as such, might be deleted at whim.

On websites with little content and just a few images, DreamHost might not care. But as long as those images start to populate the wp-content directory, there is a real danger that it can be considered a backup and deleted.

Needless to say, this makes me have serious doubts about how I can continue to use DreamHost’s services in the near future. My issue now is that, unlike before, I cannot sleep safely any longer — at any moment, I might get yet another site flagged as “backup” and terminated. It means that, except for low-traffic, low-content websites, with few (if any!) multimedia files — which should be safe to keep around — all the others are at risk of being deleted.

I’m actually quite glad that the only corporate customer I have is my own company, which has few customers and even less traffic. It means that the non-profits, at least, will understand things like temporary deletion of all their content — it’s not as if they are actually paying anything. But I can imagine that other DreamHost customers, who are used to hosting their clients’ websites here, might now suddenly be at risk from preemptive deletion, without appeal.

Note that I’m not complaining about DH’s terms of service, acceptable use policy, or their interpretation of what is “backup” and what is “content”. When signing up to an American service, I’m aware that companies can change the rules at will and demand customers to comply or have their service terminated. This is acceptable in terms of business in the USA. If you dislike that kind of policy, you should avoid hosting in the US. I host with DreamHost for so many years because, so far, I have only seen technicians being very reasonable about their demands, and very helpful in explaining how to fix things. In return, I have been quite tolerant with service interruptions and catastrophic failures, and patiently remained faithful to DreamHost. And will continue to do so.

But I’ve learned an important lesson: it’s time to make sure that any website with a lot of content, and which requires a reasonable guarantee that it won’t be deleted at whim, cannot be hosted at DreamHost any more. I had no choice but to move them to Europe, where consumer protection laws disallow such behaviour (aye, it also means poorer access for US visitors…).

Not so long ago, I already had to move a few selected sites out of DH just because their technicians weren’t willing to fine-tune the physical server where they were hosted, and performance was a nightmare. After all, DH does not guarantee any kind of performance on their shared services — that’s why they offer VPS as an alternative. Ironically, I just found out recently that one of the major reasons for poor performance was that a lot of sites on that server still ran PHP 5.2. Just upgrading 30+ of them to PHP 5.4 FastCGI brought the CPU load to single-digit load average. Astonishing, but true.

The lesson learned, to recap:

  1. Don’t host large sites on DreamHost. “Large” means anything with lots of multimedia — videos, music, PDFs, even lots of images. Stick to low-content sites with minimalist design and images. Then you should be safe.
  2. Always use PHP 5.4 FastCGI.
  3. Forget backups. Forget, in fact, anything that DH offers as backup services. It’s simply too expensive. Just get a cheap hardware server (not a VPS!) with a couple of 0.5TB disks somewhere in France, UK, Germany, or the Netherlands and install something basic on it (it could just be a humble SFTP server…). It’ll cost little more than a DH account, but you’ll have your full-fledged server for that amount. Of course it will be unmanaged for that price.

Why stick to DreamHost, then?

  1. Managed service. It’s awesome to have someone around to do the dirty work when everything fails.
  2. Technical support. Except for discussions about their policies, DH’s support is beyond extraordinary. You don’t simply get answers to your requests; you get complete performance fine-tuning instructions, often step-by-step. It’s an education on its own!
  3. Awesome control panel. The more control panels I see from their competitors, the more I like DH’s. It’s still unbeatable in terms of ease of use and incredible flexibility and options.
  4. No billing fusses. In almost 8 years, I had not a single billing issue.
  5. Hilarious newsletter. Ok, I know most people won’t care for that, but I truly laugh every month or so, when I read it!

#2

I appreciate your massive and comprehensive essay here. This potential DH user (considering migrating from current host) thanks you greatly.

–Bob


#3

After being with Dreamhost for ten years they sent me a notice that my data was no longer allowed on the server side. In one day they stripped my 100 gb of data. It can no longer be accessed where I have things displayed… on forums and such. I used it all the time to access my stuff. TEN YEARS


#4

Looks like I need to research a new host… just in case DH pulls this on me.


#5

This just happened to me. I got a “prohibited usage” email and DH has started migrating a bunch of files to dreamobjects. I’ve been with DH for 11 years. I use ftp for client and subcontractor access to files. This was never a problem before. Did the ToS change recently? It’s really a shame because to keep doing things the way I have been, I need to upgrade to a DH dedicated server package at 10x the cost. I guess I need to look elsewhere, sadly. Any suggestions? I really just want to be able to keep my website hosted in the same place as where my clients have secure ftp access to upload/download files that we’re using for current projects.


#6

They did the same thing for me. They said my video files violate their Terms of Service and are considered “archives.” (None of the files were up on my website for more than a few days and were so my client could view the work).

Their “Unlimited Storage” is a load of BS. They’ll find a way to charge you extra for having more than a few GBs.
[hr]

They did the same thing with me. They didn’t care when I threatened to leave for a different host, so I’m in the process of migrating all my data to another host.


#7

This is a really interesting discussion. It would be really nice to get DH’s official reply here.

Reading between the lines of some of the posts, it seems that most of the cases deal with a lot of content which may not be accessible from an active website. If a directory or user contains a CMS install filled with content but it’s not actually reachable from a hosted domain, then that would be considered a backup in my opinion. I believe the 50GB backups user is a good place for that stuff and I try to offload my logs, old media files, database dumps, etc there.

The one troubling example is the media files of a discussion board which were deleted. If these were all files actively linked from a working discussion board, I’m not sure how that would have been justified. Of course if the discussion board was being used primarily as a means of sharing files, then that would violate the part of the ToS about websites which primarily exist to allow the public to consume the server’s resources. There must be thousands of discussion boards at DreamHost where the line between media-heavy and media-sharing is not clear.

I’m sure DreamHost is starting to crack down on edge cases and grey-zone usage of their shared hosting in order to monetize their DreamObjects service. So the practice of sharing large files via FTP which were never meant to be accessed through an active website may have been overlooked in the past, but it probably always has technically been a violation of the ToS.

In short, it’s impossible to make any judgement without hearing both sides of the issue. DreamHost should comment to provide some balance. It would also be great if they tried a bit harder to work with those they consider abusing the unlimited policy rather than taking unilateral and unannounced action such as deleting files.
[hr]

[quote=“Gwyneth_Llewelyn, post:1, topic:61082”]
In more recent times, DH has updated their terms of service and acceptable use policies, without my noticing. I know that’s my fault, not DH’s. Now it’s strictly forbidden to make backups.[/quote]

I appreciated your very lucid account of your experiences. It seemed fair and honest. I wonder, though, which specific clauses you are referring to here. What do you remember them to be compared to now? Based on your post, I went back and re-read the ToS, AUP, and Unlimited Policy and they appear to be almost the same as when I read them in 2008ish. And nowhere does it say that backups are strictly forbidden, at least not in the way that they are strictly forbidden on other sharded hosts I have looked into.

The AUP still states that account resources should be primarily used for running a website. That does give some wiggle room for VCS repositories (related to actual sites), archived websites, etc as long as the ratio of that extraneous stuff is low compared to the stuff which is actually accessible via an actively hosted domain. Other hosts state that any files which are not publicly accessible via a URL are forbidden, so I think DH’s current wording is a nice compromise between ultra-strict and so lenient that people backup their photos and music libraries to their accounts.

In your case, was the cumulative disc usage of your active, accessible sites at least an order of magnitude larger than the space taken up by inaccessible content and archived sites? Were you making use of the 50GB backups user space?


#8

First, I’d like to apologize for the lack of clarity in previous/current policies and for the sense of distrust caused by this situation. The reasoning behind our actions should have been explained to you (Gwyneth, and others affected) a little better than it seems to have been, from what I’m reading here. We are also open to discussing these cases further if we have clearly mistaken the circumstances and you’re willing to provide us with some additional detail on how the data was being used.

As bobocat was alluding to here, the issue we have is with content on the web server not intended for public access. All data uploaded to and stored on the web server needs to be directly accessible by the public via your hosted site(s). This does mean that your site cannot be used to share files exclusively between limited parties. Yes, there may be some grey areas per our current Unlimited Policy; we manually review these situations and can discuss them further when new information is presented.

I understand any related assumptions; while it certainly does relate, we are not suddenly doing this to ramp-up use of our DreamObjects storage service. This has mainly been an effort to ensure as much of our shared hosting resources as possible can be dedicated to the hosted sites on the servers. Stability of services is our first priority, and we do typically look at data a little closer on servers that look to be filling up disks too quickly.

We do most definitely work with anyone affected by this policy enforcement. We (again) manually review each case that comes back to us before making “final” decisions on where the data should be kept. There have indeed been cases where upon further review, our actions are reversed and the data in question is moved back to the live site directory.

As discussion forums go, just to be clear, we would not want to needlessly shut down an active forum or remove any media without good cause. A board specifically for image sharing may be a different story, as bobocat mentioned above.

As a reminder, the 50gb / “backups” user is no longer a current DreamHost offering.


#9

Wow, I hadn’t noticed that until you mentioned it. Existing customers will still be able to use it or will it eventually go away like files forever, perhaps after the next crash of the server?


#10

Wow. When was this rolled out if I may ask? Does it also apply for both existing and new customers?


#11

This was rolled out about a month ago. It applies mainly to new customers, but the formal “End of Life” period for existing users, while around the corner, is not quite here yet and may happen as late as next year. We are still discussing the best processes for continued storage of the data still there— more details will be announced as we get closer to completely retiring the service.


#12

Is it safe to assume that there will be some equivalent amount of storage offered in lieu of the 50GB for customers which have paid for the service well in advance? For me, the backups user was one of the primary factors in my decision to pay for multiple years in advance.


#13

…the 50gb backups was the top reason I would get people to sign up with DreamHost hosting.
The retirement of this service, combined with the problems I have had with VPS and shared hosting on DreamHost, have motivated me to find different hosting.

With the retirement of the backup account, I will start migrating the various DreamHost hosting accounts I manage for people.

I do like DreamObjects…works well for me, so there is that at least.


#14

It’s not a very good replacement for the backups user account though. From my superficial reading of the DreamObjects entry in the wiki, there’s no way to mount backups stored in DreamObjects buckets as you can with sshfs. The killer app aspect of the backups user is being able to just [font=courier]sshfs b999999@hanjin.dreamhost.com ~/mnt/db_backup[/font] and access/manipulate files directly. No hassle at all. you just extend your file system by 50GB in California. Backup and archive cron jobs from DH regular user accounts is just as easy.

Personally I’ve never understood the use case for DreamObjects, but I’m sure many people never took advantage of the backups user because of the restricted shell it used. For me, it’s perfect, especially for avoiding violating the acceptable use policy on unlimited storage (to relate the topic back to OP)


#15

Sounds like you aren’t aware of s3fs. Same idea as sshfs, except it accesses data over S3 instead of over SFTP.


#16

Yep. s3fs works really well. I use s3fs-fuse:

I have my buckets mounted permanently in my fstab. Extends my file-system by as much as I can possibly want for just a few cents per GB.
I can access the files directly. I use it for daily backups on all servers I run, as well as hourly backups while developing.

supports chmod chown and rsync

I must admit, it actually is quite a nice feature.


#17

Nope, I’m not aware because I’ve never had the need for any cloud-based storage because I have a 50GB backups user account! :slight_smile: