Fresh discussion, using DO for backups

dreamobjects

#1

For an average home or small business it helps to have client software that facilitates scheduled backups and adhoc recovery of specific files.

What solutions are out there for end-users to make use of DreamObjects? I’ve seen the client app page on the wiki. It’s pretty sparse. And I know we can use Cyberduck but that’s more of an administration tool than a scheduled backup solution.

I’m impressed with what I’ve seen on the ownCloud site but I haven’t yet loaded the client or wired it to a DH/DO bucket. Is that all that’s required or is the ownCloud server software required as a proxy to bucket space? And can we load the OC browser UI/server in shared DH space? And would that client be considered an end-user solution?

Many of you are familiar with backup solutions that allow end-users to select folders for syncs, allowing file exclusion/inclusion based on name or regexp patterns. Many of these allow for multiple profiles which get triggered at different times. Is anyone using something like this with DO? SOS Backups come with some excellent software for local and remote backups. I’m hoping to find something like that for DH/DO.

One approach to backups would be to have a local PC backup selected files to a staging area, maybe on another partition or network drive, and then schedule complete dumps of that area to a cloud repository like DO. Does anyone do that? While not very user-friendly, it seems to me anyone could use WinMerge or a similar utility to diff their local file space, have that app copy/overlay a middle-tier, and then use CyberDuck or similar to scrape that tier off to remote storage. But it sure would be nice to automate that with scripting.

TIP for DH: If you folks could assign any resources to defining a solution like this, you’d eliminate that burden and probably get more people to pay for the storage. That’s how these other companies get consumers - they make sell a product, not just a platform. I appreciate that you’re providing a platform for your clients but it can’t hurt to help us productize it, even with a recommendation to FOSS that we can build upon a a standard.

Thanks.


#2

Hello Starbuck!

I’ve started using Syncovery.

The trick is to select the “Amazon S3” protocol and enter
"[bucket name]@objects.dreamhost.com" for the “Bucket.”

It works very well for me, YMMV.

You can try it for thirty days (I believe thirty days of use, not thirty calendar days).

Good luck.


Mamerto
[hr]
(Forgot to say.)

If you’re on a Linux desktop, you can try Syncovery’s Linux implementation, Super Flexible File Synchronizer. It does not (yet) have all the bells and whistles of Syncovery, but its free and works very well as well.


Mamerto


#3

I’m a bit off topic but I think DreamObjects is not intend for backup.
You can use it as backup storage but it’s not designed to be.

The pricing model of DO also reflect this
DO charge for storage at maximum value at any given time in a month.

When you make a schedule backup you are likely to design it this way:
you upload current backup and then delete your oldest backup after the upload process was confirmed; or else you may end up with huge bill.
this mean you will surely pay for 1 block of backup space without even really using it for whole month.

There are cheaper and better alternative for backup out there.
In your case I think you should check Amazon Glacier out.
And maybe combo with Dropbox for hot data backup. (Glacier is cold one.)
I think it designed just right for your need and could lower your bill.

And of course Dropbox is super easy for end user. (also can privately share directory with your system’s DB account.)
You can make your own background system to take snapshot to Glacier transparent to user.


#4

Hello JohnnoBravy.

I respectfully disagree.

Not that storage pricing is base on maximum monthly usage, but that OD is not suitable for back-ups.

Syncovery and many other back-up utilities can be configured to only load the portion of the file that changed or will delete a changed file before loading the newer file.

Hence, the uploading first and deleting after backup scenario is not necessarily the case (unless the particular back-up utility you use is designed to work that way).

Unless you don’t touch the files for three months, Glacier will cost the same as S3. In addition, I believe that 32 KiB of metadata are added by Glacier to each file before storage.

I’m not familiar with Dropbox but would doubt that it charges less than $0.05/month per GiB of storage.

Unless the OP intends to store terabytes of files, DO prices are very attractive (actually cheap) for storage purposes.


Mamerto


#5

One free option is Duplicity. Here’s a post talking about how to use it with DreamObjects:
http://www.dreamhost.com/dreamscape/2013/02/11/backing-up-to-dreamobjects-with-duplicity/

This is another post talking about S3, but it will also work with DreamObjects (and it additionally mentions a simpler way to use Duplicity):
http://blog.phusion.nl/2013/11/11/duplicity-s3-easy-cheap-encrypted-automated-full-disk-backups-for-your-servers/

Those approaches may not work well under Windows, but might work if you had a local backup server on your network running either Linux or Mac OS X.


#6

@mamerto

In that case it sync scenario.
Sync can happen any time but I think (guess) the OP just mentioned about syncing because he try to simplify his explanation.
However, his real intention is to make schedule backup. (happen at predefined interval and probably keep more than 1 snapshot)

And depend on how you use it Glacier is considered cheap for backup. (At least in my opinion)
Because typically you don’t need backup until something happen to your origin data.
so typically you just pay for Storage+UploadTraffic+UploadRequest+DeletionRequest which is
$0.01/GB/month , $0, $0.00005/request, $0 respectively

I’m not sure about how he use it so I just suggest him to check it out.
Maybe you are right
but if it is schedule backup I just guess he will store archive file (e.g. .tar.gz)
and also probably keeping for at least 3 months.

DreamObjects definitely competitive for some type of backup.


#7

Followup from the OP…

@Dallas, thanks for mentioning Duplicity but it’s not a consumer utility by any means. Even for developers it seems too complex. And it’s not Windows-friendly, so it’s of no use to most home users. All of these factors force it off the table.

@JohnnoBravy and @Mamerto, I need to spend more time evaluating your responses. I really want an effective backup solution for typical Windows users. Everyone needs to follow the same pattern: upload a baseline of files, then periodically upload updates, restore only as required, clean up old copies as desired. It doesn’t matter if the updates are via deltas or if there is an update/replace operation being performed. The bottom line is that the remote server should automatically reflect the user’s PC - much like Ceph uses CRUSH to replicate across DO repositories.

The situation @Johnno describes is real and concerning, where we want to ensure that we always have good backups, so we need to create a new file in the remote server before removing the original. That sort of shuffle is an unreasonable burden even for sophisticated users. However, if cycling/versioning backups then we might have completely separate buckets for each version. So we’ll have a MondayMyPC bucket and a TuesdayWifesPC bucket, etc. Again, this is an administrative burden, and it seems to be quite costly. even at the low DO price.

In the “you get what you pay for” department, we see that Glacier and perhaps other raw providers might end up being better solutions for this problem, even at more than twice the price of DH. Then we graduate to products like Carbonite, Barracuda, or SOS, where the price is more than 10x, but they (let’s say SOS in particular and the others “maybe”) store infinite versions of data at a minimal price, they only upload deltas, eliminating bandwidth and related charges, they come with very good software, and they’re already “industry standards” and very popular and consumer friendly solutions.

As to software, yes, Syncovery looks pretty good, but I’ve seen many like it that looked good but somehow didn’t stand the tests of time. For the price it Better be good! And that said, is this really a good pairing with DO?

So I’m trying to give DO a chance here. DreamHost tech and management people, toss me a bone here. I’m looking to build a solution around your platform, giving you an opportunity to position this as investment of yours in a way that will help you to recover the investment and earn profits. But from what we’re seeing here it’s simply not competitive even at the (sorry) deceptively low and alluring price. Help us to work through this extremely common problem - without resorting to beta FOSS and skills that few people other than Kyle possess.

Looking forward to more discussion and suggestions on this topic! Thanks!!


#8

I couldn’t agree more. I signed up for cloud storage over a year ago thinking “finally a solid way to maintain reliable backup that while off-site is still in the same place as all my web stuff” My 1st excursion was with a group of brits called mypcbackup I think. The software was simple enough, I told it what I wanted backed up, and off it went without any more input from me. It took the dam thing almost 6 months to backup probably 1.5 to 2 TB of data but once uploaded it was done and I felt more secure than I have in a while about not losing data. Then they contacted me and told me that I had reached their unlimited storage limit and wanted me to use a commercial plan for a staggering amount of money. I griped about the irony of their unlimited storage limit, they refunded 1/2 of my money, and I was back to square one. They’ve been contacting me ever since inviting me to come back, they haven’t erased me yet. So the dreamhost cloud thing seemed likely the backup solution of my dreams.

Now I don’t understand half of what Starbuck is saying in his two posts here but I can tell he has hit the nail on the head. For the life of me, I haven’t been able to figure out how to get my 3TB mirrored RAID DATA drive backed up to my Dreamhost Cloud Space. Two software programs DH originally recommended were totally confusing and ultimately just sat there, not uploading a thing. How hard could it be to write a simple backup routine and make it a one click install even - impossible for me but a cakewalk for the cyber geniuses at Dreamhost Come on Boys (and Girls) help us out and make gobs of money at the same time - geeze, sometimes you cant GIVE your money away.


#9

First, THE GOOD NEWS:

OK, thanks to @mamerto who provided the bucket address and suggested Syncovery I was re-inspired to actually use my brain and an old skill I learned as a child called reading and what do you know?! Syncovery is now happily buzzing away, doing my bidding.

For anyone out there who is as lazy as me (probably not possible), to run Syncovery as a basic backup program you’ll need to input all of the following when you get to the Right-Hand Side box:

Right-Hand Side: [don’t type anything] click on the circle (world globe) to the right of the browse button

In the multi-tab box (called Internet Protocol Settings), on the first tab (settings) click the down arrow in the selector box at the top (Protocol:) <-- all I did was to close a parenthetical statement ending with a colon and I got this smiley so beware, this is one of the dangers of the modern internet) that has FTP preselected. This is where you get to pick the “Amazon S3” option everybody talks about.

Now fill in the 4 boxes:

Bucket: [bucket name]@objects.dreamhost.com

Reduced Redundancy - I have no idea whether to check this checkbox or leave it alone but it sounds good and the default is “checked”

Folder: [I left it blank but presumably it would be a subdirectory]

Access ID: [this is where you enter the “key”]

Secret Key: [this is where you enter the “secret key”] - look at your dream object configuration page, its to the right of the “key”

really dreamhost (or amazon probably but DH used their protocol)??? a “key” AND a “secret key”??!!? hint: you’ll have to send in 3 crackerjack box tops to get the vitally important “SUPERSECRETKEY” (of course you’d have to be as old as me to even know what a crackerjack is; and no, it’s not X-rated)

For the rest of this tab you are on your own but from what I can gather from Syncovery’s help file you definitely want to check “Filename encoding:” at the bottom if you don’t want to lose the date/time stamps on your files. Syncovery will rename the files back to original with appropriate timestamps when you restore/retrieve the files.

Click “OK” at the bottom and then “next” and you will be taken thru a series of typical backup questions. I would suggest the “mirror” option to start unless you are seriously OCD about saving old versions of files.

Now, THE BAD NEWS

In my case, as with the Brits, Syncovery is predicting 60 days to complete the upload of my 1.4 TB and I’m getting the sinking feeling that DreamHost will be taking my home, wife, and kid away from me before its over. I make music videos for You-Tube that make no money which accounts for the large chunks of data. I nevertheless intend to go thru my data drive and see how much superfluous junk I can delete. The joke’s on Dreamhost anyway; I don’t have a home, a wife, or a kid!


#10

Hey I’m only 36 and I know what a crackerjack box top is for! :wink:

Secret key and key is … not our fault. http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSGettingStartedGuide/AWSCredentials.html

IIRC they were originally public key and private key, but that was confusing too :confused:

DH is 4cents per gig of transfer, and transfer in is free, so … this may not be TOO bad.


#11

At 36 you are truly a babe in the woods. You may want to be a little more discreet about just how vast your experience is already ;^]


#12

Nah, most people know I’m in my mid-thirties. It’s all about how you USED that time :slight_smile: