DownloadFile with thousands of files is blocking the application interface

Question

I am downloading 15019 images in JPG format with this code:

for (const item of requestUrl) {
    const optios = {
        fromUrl: item.urlPath,
        toFile: item.path,
    };
    downloadFile(optios).promise;
}

❌ The app interface is getting blocked

Note important: I know that RN runs on just one thread, so I implemented Headless JS with running in the foreground and background, it's working, but with the same behavior, the interface is blocking.

Can anyone help me solve this problem?

react-native-fs": "2.13.3

react-native: 0.59.9

score 4 · Accepted Answer · answered Aug 22 '19 at 01:26

react-native-fs does all the work in it's native implementation and exposes an async api through the react-native bridge. Therefore your js Thread is idle most of the time while downloading files through react-native-js and it shouldn't matter whether your js is running in the foreground or background.

Actual Issue

The actual issue with your implementation is that downloadFile will trigger the download and return a Promise instantly. Therefore your for loop will run through all the 15k iterations almost instantly, starting 15k concurrent downloads in the native part of your app. This is probably enough to slow down any device that much that the whole app is blocked.

Solution

If you need to download many files, you should implement some sort of queue and pipelining to limit the number of concurrent downloads. (Even if the device could handle 15k concurrent downloads without issues the download is limited by the available bandwidth. Having ~3 concurrent downloads is probably enough to use the maximum bandwidth all the time.)

A possible pipelining solution could look like this:

(The following code was not run or tested)


/**
 * 
 * @param {item} The item to download
 * @param {item.urlPath} The url to download the item from
 * @param {item.path} The local path to download the item to
 * @returns the Promise returned by react-native-fs downloadFile
 */
const downloadItem = ({ urlPath, path }) => {
  const options = {
    fromUrl: urlPath,
    toFile: path
  }
  return downloadFile(options);
}

/**
 * 
 * @param {Array} Array of the items to download
 * @param {integer} maximum allowed concurrent downloads
 * @returns {Promise} which resolves when all downloads are done
 */
const downloadPipelined = (items, maxConcurrency = 3) => {
    // Clone the items Array because we will mutate the queue
    const queue = [...items];
    /**
     * Calling this function will
     * - donwload one item after another until the queue is empty
     * - return a promise which resolves if the queue is empty and
     *   the last download started from this workerFn is finished
     */
    const workerFn = async () => {
        while (queue.length > 0){
            await downloadItem(queue.shift()); 
        }
    }
    /**
     * starts maxConcurrency of these workers concurrently
     * and saves the promises returned by them to an array
     */ 
    const workers = Array(maxConcurrency).fill(workerFn());
    /**
     * will resolve when all worker promises have resolved
     * (all downloads are done)
     * reject if one rejects
     */
    return Promise.all(workers);
}

score 1 · Answer 2 · answered Aug 22 '19 at 00:06

are you using some kind of state management? Redux, context, redux-saga/redux-thunk? You can use react-native-background-downloader to try and download through background.

If you are using redux and redux-saga you can download it using fork so it would be non-blocking.

Also, if you are downloading images and storing it or it's path in props or state, it probably is rerendering for each image downloaded.

DownloadFile with thousands of files is blocking the application interface

Note important: I know that RN runs on just one thread, so I implemented Headless JS with running in the foreground and background, it's working, but with the same behavior, the interface is blocking.

2 Answers2