How to send huge data (1 million size array of objects) from one service to another service in Nodejs (microservice)?

Question

I am using microservice architecture. Let say there are two services i.e A and B. I am trying to request from service A to service B which fetch some data from database and give that data as response to service A. But when there is huge amount of data then service B is unable to send in response but it prints on console. I tried many things but none were worked. Please help me on this.

SERVICE A

function makePostRequest(url, data, cb) {
    let postContents = {
        headers: {
            'content-type': 'application/json'
        },
        url: url,
        form: data,
        timeout: 1200000
    }
    console.log('POST ==> ', postContents)
    request.post(postContents, function(err, response, body) {
        console.log(err)
        if (err) {
            return cb({
                code: httpStatus.serverError,
                message: err
            })
        } 
        else if (response.statusCode != 200) {
            return cb({
                code: response.statusCode,
                message: body
            })
        }
        else {
            console.log(body, 'bodybodybodyPOST')
            try {
                var data = JSON.parse(body);
                if (data && typeof data == 'object')
                    return cb(null, data)
                else
                    return cb({
                        code: httpStatus.serverError,
                        message: 'invalid response'
                    })
            }
            catch(Ex) {
                console.log(Ex)
                return cb({
                    code: httpStatus.serverError,
                    message: Ex
                })
            }
        }
    })
}
exports.myapi = (req, res) => {
    makePostRequest(SERVICE-B-URL, POST-DATA, (e, d) => {
        if (e) res.status(500).json({msg: 'please try later'})
        else res.status(200).json({msg: 'data fetched', result: d})
    })
}

SERVICE B

console.log(result)
console.log('sending response ...', resTotal)
res.status(200).json({
                    total: resTotal,
                    result: result,
                    condition: req.query
                  });

[ RowDataPacket { f_stamp: 2019-05-25T05:17:48.000Z, f_player_id: 33370333, amount: -0.5, f_param_notes: null, f_money_type: 'R', f_type: 84 }, RowDataPacket { f_stamp: 2019-05-25T05:14:44.000Z, f_player_id: 31946955, amount: 30.9, f_param_notes: null, f_money_type: 'R', f_type: 70 }, RowDataPacket { f_stamp: 2019-05-25T05:14:41.000Z, f_player_id: 31035703, amount: 258, f_param_notes: null, f_money_type: 'R', f_type: 70 }, ... 163783 more items ] sending response .... 163883

send data in chunks. send entire data at once will hang create more issues. — mehta-rohan, Jun 06 '19 at 13:12

score 0 · Answer 1 · answered Jun 06 '19 at 13:08

0

A solution is to combine nodejs streams and websockets, here an example:

Streaming data in node.js with ws and websocket-stream

Usually when you have a big amount of data you want to fetch a little portion at time and send it. In the other end you can collect those little pieces and reconstruct the bigger data.

With a single HTTP request, you usually can a 413 error (Payload too large). In order to avoid this error you should setup the server to accept bigger payloads.

ExpressJS example (more info here: Error: request entity too large)

app.use(express.json({limit: '50mb'}));

Nginx example

server {
    client_max_body_size 100M;
    ...
}

answered Jun 06 '19 at 13:08

Andrea Franchini

456
3
12

I am getting this error: ```Error: Quit inactivity timeout``` – Arjun Singh Jun 07 '19 at 04:52
It isn't an http error. Seems to be related to mysql. Here more info [link1](https://github.com/mysqljs/mysql/issues/1223#issuecomment-142765722) - [link2](https://forums.developer.amazon.com/questions/105045/error-quit-inactivity-timeout.html) – Andrea Franchini Jun 07 '19 at 06:22
i have handled that error but ***service A*** is unable to receive data after ***service B*** sent the response – Arjun Singh Jun 07 '19 at 06:41
With handled do you mean catched? We should make node don't raise any error. It seems like service A close the connection for inactivity cause the long time taken by himself to send the request, and by service B to receive it (until is fully received service B won't send a response). Have you tried the streaming/socket setup? – Andrea Franchini Jun 07 '19 at 06:59
1

sending data in chunk solved my problem. But thanks, learned something new ```streaming/socket``` – Arjun Singh May 07 '20 at 20:48

How to send huge data (1 million size array of objects) from one service to another service in Nodejs (microservice)?

1 Answers1