streaming from large files and creating an array

Question

I'm having problems with highland.js. I need to create an array of functions from my stream data, but can't get it to work. Here's my code, however requests is always empty.

var requests = [];           
_(fs.createReadStream("small.txt", { encoding: 'utf8' }))
        .splitBy('-----BEGIN-----\n')
        .splitBy('\n-----END-----\n')
        .filter(chunk => chunk !== '')
        .each(function (x) {

            requests.push(function (next) {
                Helpers.Authenticate()
                    .then(function (response1) {
                        return Helpers.Retrieve();
                    })
                    .then(function (response2) {
                        return Helpers.Retrieve();
                    })
                    .then(function () {
                        next();
                    });
            });

        });
        console.log(requests)
        async.series(requests);

I just re-read your question. Mind telling us what `async.series(requests)` does? But in general, if you're expecting to `console.log` the requests as seen on the line above `async.series` then of course it's going to return empty cause streams don't block just like async. — shriek, Sep 23 '16 at 14:36
This is async.js (https://caolan.github.io/async/docs.html#.series) library — user1513388, Sep 23 '16 at 14:39
I see. So it's unrelated to the issue you're facing. You probably need to create a promise or pass a callback for the stream to finish populating `requests` and then `console.log` out or perform `async.series` on it. — shriek, Sep 23 '16 at 14:42
.each is a jQuery function. if you are using forEach, the X argument is not used. — manuerumx, Sep 23 '16 at 14:42
Maybe im wrong. Helpers.Authenticate is an ASYNC function. EACH is a Sync method. So, you need to push inside the Authenticate — manuerumx, Sep 23 '16 at 14:46

score 1 · Accepted Answer · answered Sep 23 '16 at 14:48

Just read highland's doc. Try adding .done to your stream and console.log out the requests.

_(fs.createReadStream("small.txt", { encoding: 'utf8' }))
    .splitBy('-----BEGIN-----\n')
    .splitBy('\n-----END-----\n')
    .filter(chunk => chunk !== '')
    .each(function (x) {

        requests.push(function (next) {
            Helpers.Authenticate()
                .then(function (response1) {
                    return Helpers.Retrieve();
                })
                .then(function (response2) {
                    return Helpers.Retrieve();
                })
                .then(function () {
                    next();
                });
        });

    }).done(function(){
      console.log(requests);
    });

Wow - thank you. Worked like a charm. I was tearing my hair out trying to work it out. — user1513388, Sep 23 '16 at 14:55

score 0 · Answer 2 · answered Sep 23 '16 at 14:43

I would just use the stream events to wire things up:

var stream = fs.createReadStream('small.txt', {encoding: "utf8"});

stream.on('data', (line) => {
    var lineStr = line.toString(); //Buffer to String
    /* You code here */
})

stream.on('close', (line) => {
    console.log(request);
})

streaming from large files and creating an array

2 Answers2