Handle the download of several tens of thousands files c#

Question

I'm making a small software that download several tens of thousands files. It's not efficient at all for now because i download each file once by once and so it's very slow, and also lot of files are less than 100ko.

Do you have any idea to improve the download speed ?

    /*******************************
        Worker work
    /********************************/
    private void backgroundWorker1_DoWork(object sender, DoWorkEventArgs e)
    {
        listCount = _downloadList.Count;
        // no GUI method !
        while (TotalDownloadFile < _downloadList.Count)
        {
            // handle closing form during download
            if (_worker.CancellationPending)
            {
                _mainView = null;
                _wc.CancelAsync();
                e.Cancel = true;
            }
            else if (!DownloadInProgress && TotalDownloadFile < listCount)
            {
                _lv = new launcherVersion(_downloadList[TotalDownloadFile]);
                var fileToDownloadPath = Info.getDownloadUrl() + _lv.Path;
                var saveFileToPath = Path.GetFullPath("./") + _lv.Path;
                if (Tools.IsFileExist(saveFileToPath))
                    File.Delete(saveFileToPath); // remove file if extist
                else
                    // create directory where the file will be created (use api this don't do anything on existing directory)
                    Directory.CreateDirectory(Path.GetDirectoryName(saveFileToPath));
                StartDownload(fileToDownloadPath, saveFileToPath);
                UpdateRemaingFile();
                _currentFile = TotalDownloadFile;
            }
        }
    }

Start Download Function

    /*******************************
        start the download of files
    /********************************/
    public void StartDownload(string fileToDownloadLink, string pathToSaveFile)
    {
        try
        {
            using (_wc = new WebClient())
            {
                _wc.DownloadProgressChanged += client_DownloadProgressChanged;
                _wc.DownloadFileCompleted += client_DownloadFileCompleted;
                _wc.DownloadFileAsync(new Uri(fileToDownloadLink), pathToSaveFile);
                DownloadInProgress = true;
            }
        }
        catch (WebException e)
        {
            MessageBox.Show(fileToDownloadLink);
            MessageBox.Show(e.ToString());
            _worker.CancelAsync();
            Application.Exit();
        }
    }

You could potentially use multi-threading and concurrency to download entire batches at once. You'd have to put some though into ensuring each thread completes successfully and ensure that files don't get downloaded twice. — JD Davis, Sep 15 '17 at 13:17
`Do you have any idea to improve the download speed ?` Please show us the existing code. — mjwills, Sep 15 '17 at 13:21
You could at least tell us what protocol you are using. HTTP / FTP / SFTP / ... — Fildor, Sep 15 '17 at 13:22
[producer-consumer-dataflow-pattern](https://docs.microsoft.com/en-us/dotnet/standard/parallel-programming/how-to-implement-a-producer-consumer-dataflow-pattern) — Fildor, Sep 15 '17 at 13:29
I've edited my question , i sent my code, and i'm using HTTP — kev.g, Sep 15 '17 at 13:36
https://docs.microsoft.com/en-us/dotnet/standard/parallel-programming/how-to-write-a-simple-parallel-foreach-loop may be of interest. — mjwills, Sep 15 '17 at 13:40
`else if (!DownloadInProgress && TotalDownloadFile < listCount)` The `!DownloadInProgress` check is what is ensuring you are only downloading one file at once... — mjwills, Sep 17 '17 at 21:50

JD Davis · Answer 1 · 2017-09-15T13:26:41.493

Expanding upon my comment. You could potentially use multi-threading and concurrency to download entire batches at once. You'd have to put some though into ensuring each thread completes successfully and ensure that files don't get downloaded twice. You would have to secure your centralized lists using something like lock.

I would personally implement 3 separate lists: ReadyToDownload, DownloadInProgress, and DownloadComplete.

ReadyToDownload would contain all objects that need to be downloaded. DownloadInProgress would contain both the item being downloaded and the Task handling the download. DownloadComplete would hold all objects that were downloaded and reference the Task that performed the download.

Each Task would hypothetically work better as an instance of a custom object. That object would take in a reference to each of the lists, and it would handle updating the lists once it work either completes or fails. In the event of a failure, you could either add a forth list to house the failed items, or reinsert them into the ReadyToDownload list.

"and reference the thread that performed the download" - why? I wouldn't even use Threads, but Tasks. — Fildor, Sep 15 '17 at 13:25

Handle the download of several tens of thousands files c#

1 Answers1