Upload multiple files AND large files

dreamobjects

#1

My dev is telling me that dreamobjects will allow our users to select and upload multiple files at once from their computer to their account on our site. we can do this with a file size limit of 50mb per file, but we will sacrifice letting users upload single files that are large. I believe with those large file uploads, the limit is 2gb. How can we do both?


#2

If I’m understanding your question correctly, you’re looking for a way to allow customers to upload something to your site and then to DreamObjects.

You can allow customers to upload files from their browser directly to DreamObjects, and skipping the step of uploading to your site. This is called browser-based POST uploads and there is a sample of how to do that in this forum post - https://discussion.dreamhost.com/thread-141720.html


#3

I’ll look into this now. The use case I was explaining, was a dropbox type of function that we have for saving and managing files for projects. each user will be uploading stuff to their acc on our site. The current road blocks we have run into with Dreamobjects, is that you can only upload a single file a time, but if we implement selecting and uploading multiple file’s from a computer all at once, then we can no longer allow the uploads of large files (the largest being 2gb). Also, the multiple file selection method has a 50mb file size limit for any one individual file.


#4

I think I forgot to say, that we want chunk upload for uploading large files. I went through the forum link and he has solution (Not TESTED YET) but it is not returning a response of what was uploaded…that is needed in our case and there is no code for multiple uploads too.
[hr]
I’m still researching, as we are using chunk Upload for large files… i think multiple selection and uploads are not possible to do in chunk mode.


#5

Hey Justin, can you let me know if what I posted makes sense? Is this something I will have to switch to Amazon and possibly Google for?


#6

Hi Sadie. The example in the other forum post uses php to create a form to upload from a browser directly to DreamObjects so that it bypasses your server. There are several examples that will return the filename, set an identifier in the metadata, even multipart uploads. Here are a few that might be helpful:

http://birkoff.net/blog/post-files-directly-to-s3-with-php/
http://blog.fineuploader.com/2013/08/16/fine-uploader-s3-upload-directly-to-amazon-s3-from-your-browser/
https://github.com/z-x/amazon-s3-direct-post-upload
http://www.spacevatican.org/2013/7/7/direct-to-s3-browser-uploads/

As for the limitations of 50MB and 2GB, I’m not clear on where that’s coming from. Is that from the library you’re using?

I will say that you’re likely to see problems uploading very large files from a browser simply due to the stability of connections over a long period of time. The multipart upload functionality should help with that.


#7

I was told by your email support, that you all are in contact. is there any other info i can provide? i believe I have provided all the necessary details, and I have replied to you, stating why the forum link you fwd me is not a viable solution.


#8

The S3 protocol doesn’t support uploading multiple files in a single operation. This isn’t a limitation specific to DreamObjects; it’s not possible with Amazon S3 or Google Cloud Storage either. This isn’t to say that it’s impossible to have a form that accepts multiple files — it’s certainly doable! — but it will be more complex than the example which Justin posted a link to above.


#9

Ok, I just saw your replies come in. Let me look into this. Thanks for getting back with me.
[hr]
it seems dropbox uses S3 and supports multiple uploads. any ideas on what we would need to do with dreamobjects? should I refer to Justin’s articles?


#10

That’s correct that Dropbox uses S3 for storage. As Andrew mentioned, S3 does not support uploading multiple files in a single operation either. You’d have to construct your code to start a new upload for each file if you want to support uploading multiple files.


#11

to reiterate, my dev keeps saying we need chunk upload, in order to support the upload of the large files and that we will have to sacrifice this for the multiple uploads. in terms of where the 50mb or 2gb limits come from, I’ll have to ask.


#12

Just to clarify, chunk upload is the same thing as multipart upload (you’ll find more search results looking for multipart than chunk in reference to the S3 API). That breaks a larger file into smaller pieces and uploads them in parts.

That is separate from multiple uploads, where that’s just selecting more than one file to upload. I don’t know why you couldn’t do both as most S3-compatible libraries have butil-in support for multipart uploads. You’d probably just have to do a bit of work in your code to allow for selecting multiple files and uploading them.


#13

in terms of the comment you just posted (see below), you mean this is what we should focus on with our Dreamobjects? I’ll double check what limitation my dev was getting with this bc he says we can’t do both with dreamobjects, but we can with S3. Maybe I’ll ask how he will do that with S3 and let you know.

That is separate from multiple uploads, where that’s just selecting more than one file to upload. I don’t know why you couldn’t do both as most S3-compatible libraries have butil-in support for multipart uploads. You’d probably just have to do a bit of work in your code to allow for selecting multiple files and uploading them.


#14

If you can do both with S3, then you should be able to do both with DreamObjects. DreamObjects uses the same API, although with a limited subset. I don’t know any limitation that would stop you doing both chunk/multipart uploads and multiple uploads using DreamObjects.


#15

Ok, I’ll update my dev then. what is the main difference with the Dreamobjects API and the S3 one. You mentioned that it uses a limited subset. Let me ask my dev, what specific limitation he has with doing both chunk/multipart uploads and multiple uploads using DreamObjects. Until then, if you have any hints to pass along, please let me know.


#16

his reply:

After wasting two days on this, I came to the conclusion below.

DreamObjects doesn’t support a direct upload to the s3 server, which is needed in our application. i am getting the error message below.

User-specified metadata

<?xml version="1.0" encoding="UTF-8"?>AccessDeniedPolicy check failed, variable not found: x-amz-meta-uuid

#17

can you send some sample code to guide me?
[hr]
the sample code that would help, would be for how to add custom meta while uploading files directly to the server.


#18

Hey guys, do you think this would be possible to get?


#19

Hey guys, can you help us get past this error we are getting below? is there any sample code or guidance you can send for how to add custom meta while uploading files directly to the server?

DreamObjects doesn’t support a direct upload to the s3 server, which is needed in our application. i am getting the error message below.

User-specified metadata

<?xml version="1.0" encoding="UTF-8"?>AccessDeniedPolicy check failed, variable not found: x-amz-meta-uuid

#20

For what it’s worth, I’m doing MultiPart uploads in the (in dev) new version of the DreamObjects WP plugin. Works like this in PHP:

$s3 = AwsS3DHDO::factory(array(
	'key'      => ACCESSKEY,
	'secret'   => SECRETKEY,
	'base_url' => 'http://objects.dreamhost.com',
));

$uploader = UploadBuilder::newInstance()
	->setClient($s3)
	->setSource(SOURCEFILELOCATION)
	->setBucket(BUCKET)
	->setKey(NEWFILENAME)
	->setMinPartSize(25 * 1024 * 1024)
	 ->setOption('Metadata', array(
	        'UploadedBy' => 'DreamObjectsBackupPlugin'
	))
	->setOption('ACL', 'private')
	->setConcurrency(3)
	->build();
					
try {
	echo 'Begin upload. This may take a while (5min for every 75 megs or so).';
	set_time_limit(180);
	$uploader->upload();
	echo 'Upload complete';
} catch (MultipartUploadException $e) {
	$uploader->abort();
	echo 'Upload failed: '.$e->getMessage();
}

The “setOption(‘Metadata’, array(” part is where I add my own Meta Data. That said, MAKE SURE you’re using at least v 2.6 of the flippin SDK because before that it was broken in a daft, undocumented way that made me want to drink.