Different ways to upload a file?

Question

I have read several ways to upload files to server.

• HTTP file uploading as multipart/formdata (How does HTTP file upload work?)

• JSON file uploading (How do I upload a file with metadata using a REST web service?)

Is there other options? The project I am working on needs such function. And my file to upload can be as big as several gigabits. I'd like to get a thorough evaluation of different approaches.

ADD 1

Thanks for so many responses. I am occupied for some other things these days and can't get back to this question until I see the SO notifications. I'd like to apologize for adding details so late as the bounty is about to expire.

In my scenario, there're 1 web server, a file server and many agents. The whole picture kind of looks like this:

• User interacts with Web server through browser for routine operations.
• User uploads files to file server through browser. (I want everything happen in browser to make customer life easier.)
• Agent is desktop applications that communicates with web server and file server.
• Agent gets routine information from web server.
• Agent pulls files from and pushes files to file server. And notifies web server about everything.
• Web server pulls files from file server to present it to customer in browser.

And some coding context:

• Java
• Spring
• Netty

Answer 1

Others ways to upload files to server (summarized):

• FTP: (File Transfer Protocol). It is a standard networking protocol that separates control and data iteration (2 different ports). Because of its insecurity, it can be used SFTP instead (linux systems). https://en.wikipedia.org/wiki/File_Transfer_Protocol

• SCP: (Secure Copy) which will allow you to transfer files via ssh protocol. https://en.wikipedia.org/wiki/Secure_copy

• WebDAV, which is an HTTP extension which allows to update content remotely from a client. https://developer.mozilla.org/en-US/docs/Glossary/WebDAV

• GitHub: which whill allow uploading files using git with a combination of commit/push methods. https://github.com/

• Rsync: incremental file transfer usually found in Unix-systems. It allows to copy only the changes from the file in the client to the server, saving bandwidth. https://rsync.samba.org/

Finally, the following link shows an implementation method for HTTP upload for Classical ASP: https://support.microsoft.com/en-us/kb/299692

This is a summary of some of the techniques or methods but there are plenty of solutions.

Answer 2

You do not explicitly specify what platform/language you are targeting. I fear it's not a web application for start, as the two options you have mentioned are most widely applied over web applications.

Well, upload of files as large as you have mentioned would need to be designed very carefully. In middle of uploading your connection could break, certain packets might get corrupted and even the uploading end-system might experience faults.

Had it been me, best bet for a safe and trustworthy file upload would be to break down the file into smaller chunks and applying a mutli-threaded program to handle upload of these chunks.At the receiving side, similar functionality for re-assembling would be needed.Advantages of this approach would be:

1. Ability to keep track of progress.
2. Resume capability could be implemented.
3. Verification via checksums will ensure freedom from corruption.

However for a precise solution, please expand your description.If you had to put in a bounty, something must have been missing in the description right!!!

Answer 3

I agree with @Ayelis comment (main post) that the question is too abstract. However I would like to add my two cents.

Actually, you have two options:

1. You want to integrate the upload process with a website and the user is not just send you a file, but also create an order/case/record (i.e. this file should be associated with some database entry and a user).
2. You need to give users an ability to transfer a file to your company even without a website.

In the first case the only option is using a HTTP-based solution. In the second case, you can try using FTP, cloud storage systems (Dropbox, ...), etc. Of course, you may use FTP in the first scenario as well, but it is viable only if you have a limited number of users (e.g. only employees).

So I assume, you are looking into the first scenario when you integrate the upload into a website.

The important limitation you mentioned is the requirement to upload several GB files. If you just add an <input type="file"> element to your page, you may encounter a number of issues (client-side memory, server-side memory, security, reliability, etc - it will depend on the choice of technologies). To address them, the best approach would be to create a JavaScript uploader which would split your file into the portions of smaller size (say, 5MB) and upload them as individual files. On the server, you can put all these files together and assembly it back to the full file. As a result:

• No need to prepare and receive very large POST requests (which may be memory-intensive) - from the technical point of view, it will be an equivalent of the upload of hundreds of regular small files.
• It is not a problem to make it reliable (if the upload of 2GB file breaks at 90%, the user won't have to re-upload it from a scratch).
• No need to turn off POST request validation rules on your server

It requires to write plenty of code both on the client and server side. However if you have a full stack developer in your team, it is not a problem. Also, you may search for a third-party solution for this issue.

If you don't want to use HTTP (e.g. the second scenario is closer to you), and feel that Dropbox/Google Drive/etc is a tempting approach, please note that for very large files it may be a poor solution (due to bandwidth and costs). I would recommend to looking at solutions like Seafile or ownCloud, at least they are free.

Hope this helps.

Answer 4

[Replying cause I find this interesting in a philosophical level!]

It is a fair question I think given the "...several gigabits". I don't think it matters what language you are trying to implement this, however, the architecture and your system requirements matter a lot (in agreement with @Ayelis). I would separate the solutions to the following categories:

1. Traditional client-server upload - I don't think this would work since a user has to leave his tab/browser on till the upload finishes

2. Middle stage/man: Solutions based on a middle stage decoupling upload and store/download (ie: Dropbox, Google drive, etc...). In this case your web app accepts a URL to "get the file from". The questions are: can your server "pull" from a location? Should your clients require to upload files publicly/accessibly somewhere? How do you show the progress of the transfer?

3. Intranet-based solutions to synchronize/mount file-systems (rsync, internal git repo, shared cifs/nfs mounted drive)

4. More on the P2P side... a native client (anything but web) which has the ability to upload, pause and resume to given remote location. In this category I also place well known P2P/torrent apps. A possible scenario would be: the client uploads a (legit) torrent, registers the magnet link or so to the web server who is queueing it for download (kinda like ktorrent web interface used to work... not sure if it still does).

Again as @Ayelis said, a little bit more info will help you get a better answer... I just find it a challenging problem!

Answer 5

If you need a strictly browser-based solution for large files, you will have to rely on JavaScript (I could say Java Applets or -- gasp -- Flash, but I wouldn't recommend it, and it is actually desktop app in disguise -- but they are an option).

That being said, for large files, you have a very narrow set of options using the File API. There are excellent solution out there, but I love Blueimp's jQuery File Upload.

https://github.com/blueimp/jQuery-File-Upload

You should take a look at their Chunked File Upload for resumable uploads:

https://github.com/blueimp/jQuery-File-Upload/wiki/Chunked-file-uploads

This should allow you to handle very large files over flaky connections.

-- If you don't have a browser-only restriction, then there are many options discussed in the answers.

Update You have a webserver serving browsers and desktop apps. In order to avoid writing server code for each one, write for the browser scenario, which is the most restrictive (sandboxed, limited access to local device). The desktop app should be easy to adapt to any scenario imposed by the browser.

If you are able to write your solution for the browser, the Agent app should be a breeze.

Update2

Your chart has a line between Browser and File Server. That is not quite correct. The Browser will connect to the Web Server only, and this will store on whatever backend you have (own server, database, other file servers, etc).

The Agent connection to the File Server could possibly work, but then you would probably be working via network protocols, for example (like writing a desktop app that would manipulate files over regular shared network drives).

But if you want your Agent to be as mobile as the browser, it should be treated as a browser for file uploads and server communication (so no lines between Agent and File Server)