Boto-rsync stalls at very large file

dreamobjects

#1

Please see the screenshot:

http://postimg.org/image/t1hrcfltp/

I’m transferring a very large archive that’s over 230GB xxxx.tar from my other host to DreamObjects.

It stalls at the file. It’s been stalled there for the last 10 minutes and I’m still waiting for it to resume.

Anyone has any experience with boto-rsync transferring very large files like this? Is this normal that it slows down at large files and that it virtually stalls?

What’s boto-rsync doing here? Anything wrong or am I supposed to wait?


#2

From experience, boto-rsync does not handle large files very well. One of the biggest problems is that it doesn’t support multipart uploads. I’d suggest using something like s3cmd, and specifically the sync option. http://s3tools.org/s3cmd-sync

We have some documentation in our wiki about installing and configuring s3cmd to work with DreamObjects. http://wiki.dreamhost.com/S3cmd

One modification I’d recommend is changing “multipart_chunk_size_mb” to something larger, say 50, since you’re uploading a 230GB file. Multipart uploads are limited to 10,000 parts.

Let me know if that works better for you.