10

I have a file with a pre-signed URL. I would like to upload that file directly to my S3 bucket without donwloading it first (I know how to do it with the intermediate step but I want to prevent it).

Any suggestion? Thanks in advance

Rico
  • 48,741
  • 12
  • 84
  • 107
desmo
  • 133
  • 1
  • 10
  • I believe this question to be a duplicate, but can't put my finger on the duplicate at the moment, so the answer is below. – Michael - sqlbot Feb 13 '14 at 17:10
  • 1
    @Michael-sqlbot The following may be the duplicate you're thinking of, but I think the current question and your answer cover the topic much more clearly. http://stackoverflow.com/questions/7586533/is-it-possible-to-upload-to-s3-directly-from-url-using-post/ – Eric Hammond Feb 14 '14 at 22:57

1 Answers1

16

There is not a method supported by S3 that will accomplish what you are trying to do.

S3 does not support a request type that says, essentially, "go to this url and whatever you fetch from there, save it into my bucket under the following key."

The only option here is to fetch what you want, and then upload it. If the objects are large, and you don't want to dedicate the necessary disk space, you could fetch it in parts from the origin and upload it in parts using multipart upload... or if you are trying to save bandwidth somewhere, even the very small t1.micro instance located in the same region as the S3 bucket will likely give you very acceptable performance for doing the fetch and upload operation.

The single exception to this is where you are copying an object from S3, to S3, and the object is under 5 GB in size. In this case, you send a PUT request to the target bucket, accompanied by:

x-amz-copy-source: /source_bucket/source_object_key

That's not quite a "URL" and I assume you do not mean copying from bucket to bucket where you own both buckets, or you would have asked this more directly... but this is the only thing S3 has that resembles the behavior you are lookng for at all.

http://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectCOPY.html

You can't use the signed URL here... the credentials you use to send the PUT request have to have permission to both fetch and store.

Michael - sqlbot
  • 139,456
  • 21
  • 252
  • 328