Whats the smartest / cleanest way to iterate async over arrays (or objs)?

Question

Thats how I do it:

function processArray(array, index, callback) {
    processItem(array[index], function(){
        if(++index === array.length) {
            callback();
            return;
        }
        processArray(array, index, callback);
    });
};

function processItem(item, callback) {
    // do some ajax (browser) or request (node) stuff here

    // when done
    callback();
}

var arr = ["url1", "url2", "url3"];

processArray(arr, 0, function(){
    console.log("done");
});

Is it any good? How to avoid those spaghetti'ish code?

Answer 1

Checkout the async library, it's made for control flow (async stuff) and it has a lot of methods for array stuff: each, filter, map. Check the documentation on github. Here's what you probably need:

each(arr, iterator, callback)

Applies an iterator function to each item in an array, in parallel. The iterator is called with an item from the list and a callback for when it has finished. If the iterator passes an error to this callback, the main callback for the each function is immediately called with the error.

eachSeries(arr, iterator, callback)

The same as each only the iterator is applied to each item in the array in series. The next iterator is only called once the current one has completed processing. This means the iterator functions will complete in order.

Answer 2

As pointed in some answer one can use "async" library. But sometimes you just don't want to introduce new dependency in your code. And below is another way how you can loop and wait for completion of some asynchronous functions.

var items = ["one", "two", "three"];

// This is your async function, which may perform call to your database or
// whatever...
function someAsyncFunc(arg, cb) {
    setTimeout(function () {
        cb(arg.toUpperCase());
    }, 3000);
}

// cb will be called when each item from arr has been processed and all
// results are available.
function eachAsync(arr, func, cb) {
    var doneCounter = 0,
        results = [];
    arr.forEach(function (item) {
        func(item, function (res) {
            doneCounter += 1;
            results.push(res);
            if (doneCounter === arr.length) {
                cb(results);
            }
        });
    });
}

eachAsync(items, someAsyncFunc, console.log);

Now, running node iterasync.js will wait for about three seconds and then print [ 'ONE', 'TWO', 'THREE' ]. This is a simple example, but it can be extended to handle many situations.

Answer 3

As correctly pointed out, you have to use setTimeout, for example:

each_async = function(ary, fn) {
    var i = 0;
    -function() {
        fn(ary[i]);
        if (++i < ary.length)
            setTimeout(arguments.callee, 0)
    }()
}


each_async([1,2,3,4], function(p) { console.log(p) })

Answer 4

The easiest way to handle async iteration of arrays (or any other iterable) is with the await operator (only in async functions) and for of loop.

(async function() {
 for(let value of [ 0, 1 ]) {
  value += await(Promise.resolve(1))
  console.log(value)
 }
})()

You can use a library to convert any functions you may need which accept callback to return promises.

Answer 5

In modern JavaScript there are interesting ways to extend an Array into an async itarable object.

Here I would like to demonstrate a skeleton of a totally new type AsyncArray which extends the Array type by inheriting it's goodness just to become an async iterable array.

This is only available in the modern engines. The code below uses the latest gimmicks like the private instance fields and for await...of.

If you are not familiar with them then I would advise you to have a look at the above linked topics in advance.

class AsyncArray extends Array {
  #INDEX;
  constructor(...ps){
    super(...ps);
    if (this.some(p => p.constructor !== Promise)) {
      throw "All AsyncArray items must be a Promise";
    }
  }
  [Symbol.asyncIterator]() {
    this.#INDEX = 0;
    return this;
  };
  next() {
    return this.#INDEX < this.length ? this[this.#INDEX++].then(v => ({value: v, done: false}))
                                     : Promise.resolve({done: true});
  };
};

So an Async Iterable Array must contain promises. Only then it can return an iterator object which with every next() call returns a promise to eventually resolve into an object like {value : "whatever", done: false} or {done: true}. So basically everything returned is a promise here. The await abstraction unpacks the value within and gives it to us.

Now as I mentioned before, this AsyncArray type, since extended from Array, allows us to use those Array methods we are familiar with. That should simplify our job.

Let's see what happens;

class AsyncArray extends Array {
  #INDEX;
  constructor(...ps){
    super(...ps);
    if (this.some(p => p.constructor !== Promise)) {
      throw "All AsyncArray items must be a Promise";
    }
  }
  [Symbol.asyncIterator]() {
    this.#INDEX = 0;
    return this;
  };
  next() {
    return this.#INDEX < this.length ? this[this.#INDEX++].then(v => ({value: v, done: false}))
                                     : Promise.resolve({done: true});
  };
};

var aa = AsyncArray.from({length:10}, (_,i) => new Promise(resolve => setTimeout(resolve,i*1000,[i,~~(Math.random()*100)])));

async function getAsycRandoms(){
  for await (let random of aa){
    console.log(`The Promise at index # ${random[0]} gets resolved with a random value of ${random[1]}`);
  };
};

getAsycRandoms();