Why is the zlib inflate function is not working in correct order in node.js?

Question

I am new to Stackoverflow. I am now working on a node.js program. The input is a data stream containing few blocks of data (also zipped by DEFLATE algorithm) concatenated together. My goal is use INFLATE to restore the data blocks and put them into correct orders.

My problem is, when I use while loop to do extraction of data blocks, the data extracted is not in the order I input. Why?

while (initPointer < totalLength) {
    ...
    console.log('Extracting '+rawLengthBuf.readUInt32LE(0));

    ...

    zlib.inflate(dataBuf, function(err, data) {
        if (!err) {
            console.log('Extracted '+data.length);
        }
    });
}

Output:

Extracting 18876
Extracting 15912
Extracting 10608
Extracted 15912
Extracted 10608
Extracted 18876

Please forgive me that I may not describe the situation in very clear way. Thanks.

score 1 · Answer 1 · answered Dec 31 '14 at 10:26

The async zlib methods do their work in the thread pool, so each inflate could be executed in parallel. Which inflate finishes first depends on a number of factors (e.g. CPU scheduling algorithm), so you cannot assume a particular order when you call zlib.inflate() in a loop like that.

score 1 · Accepted Answer · answered Dec 31 '14 at 11:39

Use Async.js. Below are steps to make your code work synchronously using async.js.

I ran into similar problem and resolved it using the steps below. I have replaced my function with yours. It should work. You just give it a try, documentation is inline to make you understand the code.

/*
 * need an array to run all queries one by one in a definite order
 */
var arr = [];

while (initPointer < totalLength) {
    console.log('Extracting '+rawLengthBuf.readUInt32LE(0));
    arr.push(dataBuf)
}


/*
 * Callback function for initiation of waterfall
 */
var queue = [
    function(callback) {
        // pass the ref array and run first query by passing starting index - 0
        callback(null, arr, 0)
    }
];

/*
 * Object to store result of all async operation
 */
var finalResult = {};

/*
 * Generic Callback function for every dynamic query
 */
var callbackFunc = function(prevModelData, currentIndex, callback) {
    //current file/data
    zlib.inflate(arr[currentIndex], function(err, data) {
        if (!err) {
            console.log('Extracted '+data.length);
        } else {
            // console.log(result)
            finalResult[arr[currentIndex]] = data
            //call next element in queue
            callback(null, data, currentIndex + 1)
        }
    });
}

/*
 * Add callback function for every dynamic query
 */

while (initPointer < totalLength) {
    ...
    console.log('Extracting '+rawLengthBuf.readUInt32LE(0));

    ...

    queue.push(callbackFunc)

}


/*
 * Run all dynamic queries one by one using async.js waterfall method
 */
async.waterfall(queue, function (err, result) {
    console.log('finish', finalResult)
});

Thanks very much for detailed reply. Actually I have come across async function but not realized it was solution before I ask here. :) — Frederick Li, Jan 02 '15 at 06:48

Why is the zlib inflate function is not working in correct order in node.js?

2 Answers2