aws-sdk calls don't work in a forked node.js process running on AWS Lambda

Question

I have a small node modules that calls a fork syscall to fork the running process. When I try to call the aws-sdk function Kinesis.putRecord from the child process, the call does not happen. Calling from the parent process works fine.

To illustrate, the following two code snippets only differ on whether the call to putRecord is performed by the parent or child processes.

In the first code snippet, the call is performed by the parent process, and the call works fine.

const aws = require('aws-sdk');
const fork = require('fork');

module.exports.handler = (event, context, callback) => { 
    const isChild = fork.fork();
    if (!isChild) { // Parent
        const kinesis = new aws.Kinesis()
        const record = {
          Data: "Lorem ipsum",
          PartitionKey: "123456",
          StreamName: process.env.STREAM_NAME,
        }
        kinesis.putRecord(record, (err, data) => {
            if (err) {
                callback(null, {message: 'Encountered error in kinesis', err});
            } else {
                callback(null, {message: 'Successfully ran in kinesis', data});
                fork.wait();
            }
        })
    } else { // Child
        process.exit();
    }
};

On the other hand, in the second code snippet, the call is performed by the child process, and the call is not performed. Specifically in this example the Lambda execution times out.

const aws = require('aws-sdk');
const fork = require('fork');

module.exports.handler = (event, context, callback) => {  
    const isChild = fork.fork();   
    if (isChild) { // Child      
        const kinesis = new aws.Kinesis()
        const record = {
          Data: "Lorem ipsum",
          PartitionKey: "123456",
          StreamName: process.env.STREAM_NAME,
        }
        kinesis.putRecord(record, (err, data) => {
            if (err) {
                callback(null, {message: 'Encountered error in kinesis', err});
            } else {
                callback(null, {message: 'Successfully ran in kinesis', data});
            }
        })
    } else { // Parent
        fork.wait();
    }
};

The fork module can be found here.

I'm using node.js version 6.10.3.

score 0 · Answer 1 · answered Apr 08 '18 at 06:22

0

Node.js is based on a single-threaded architecture. AWS Lambdas are designed and heavily optimized to execute relatively short-lived functions. Your lambda is most likely only going to be allocated one virtual CPU when running.

What problem are you trying to solve with fork that couldn't be solved by running two lambdas in parallel? There are many ways of triggering lambdas so that wouldn't be too hard. There's even the notion of calling one lambda from another asynchronously.

answered Apr 08 '18 at 06:22

Kim Burgaard

3,368
16
11

I'm trying to enforce stateless execution of lambda functions while minimizing cold start penalties per execution. The idea is to execute the handler from within the child process (in a separate memory address space), keeping the original node.js process clean. – Kalev Apr 09 '18 at 08:29

aws-sdk calls don't work in a forked node.js process running on AWS Lambda

1 Answers1