AWS Lambda RDS connection timeout

Question

I'm trying to write a Lambda function using Node.js which connects to my RDS database. The database is working and accessible from my Elastic Beanstalk environment. When I run the function, it returns a timeout error.

Tried to increase the timeout up to 5 minutes with the exact same result.

The conclusion I came to after some research is that it's probably a security issue but couldn't find the solution in Amazon's documentation or in this answer (which is the only one I could find on the topic).

Here are the security details:

Both the RDS and the Lambda are in the same security group.
The RDS has All traffic inbound and outbound rules.
The Lambda has AmazonVPCFullAccess policy in it's role.

My code is:

'use strict';
console.log("Loading getContacts function");

var AWS = require('aws-sdk');
var mysql = require('mysql');

exports.handler = (event, context, callback) => {

   var connection = mysql.createConnection({
        host     : '...',
        user     : '...',
        password : '...',
        port     : 3306,
        database: 'ebdb',
        debug    :  false
    });

    connection.connect(function(err) {
      if (err) callback(null, 'error ' +err);
      else callback(null, 'Success');
    });

};

The result I'm getting is:

"errorMessage": "2017-03-05T05:57:46.851Z 9ae64c49-0168-11e7-b49a-a1e77ae6f56c Task timed out after 10.00 seconds"

https://notebookbft.wordpress.com/2018/01/09/querying-rds-mysql-db-with-nodejs-lambda-function/ — Rajind Ruparathna, Jan 13 '18 at 02:16

score 29 · Answer 1 · answered Jul 17 '17 at 15:07

While using context will work, you just need to add context.callbackWaitsForEmptyEventLoop = false; to the handler and then use callback as normal like this:

exports.handler = (event, context) => {
  context.callbackWaitsForEmptyEventLoop = false; 
  var connection = mysql.createConnection({
    //connection info
  });
  connection.connect(function(err) {
    if (err) callback(err); 
    else callback(null, 'Success');
  });
};

The answer is here in the docs (took me a few hours to find this): http://docs.aws.amazon.com/lambda/latest/dg/nodejs-prog-model-using-old-runtime.html

In the section "Comparing the Context and Callback Methods" it has an "Important" note that explains things.

At the bottom of the note it reads:

Therefore, if you want the same behavior as the context methods, you must set the context object property, callbackWaitsForEmptyEventLoop, to false.

Basically, callback continues to the end of the event loop as opposed to context which ends the event loop. So setting callbackWaitsForEmptyEventLoop makes callback work like context.

Legend! thank you, this is the correct answer: context.callbackWaitsForEmptyEventLoop = false; — Gary Benade, Jun 09 '18 at 14:41
Not all heroes wear cape! thank you. context.callbackWaitsForEmptyEventLoop = false; is correct context.callbackWaitsForEmptyEventLoop = false; const response = { statusCode: 200, body: JSON.stringify({ headers: { 'Access-Control-Allow-Origin': '*', 'Access-Control-Allow-Credentials': true, }, message: await mysqlConnector.create('table_name', {test: 50}), input: event, }), }; callback(null, response); — RenanSS, Apr 29 '19 at 13:32

Vor · Answer 2 · 2021-05-26T09:42:31.403

14

Both the RDS and the Lambda are in the same security group.

That's the key. By default communication within the same security group is not allowed. And you need to explicitly allow it (E.x sg-xxxxx ALL TCP ). This wll only work if your lambda tries to access db by private ip.

If it tries to access it by public IP that it will not work and you need to punch necessary wholes for that as well.

However there is better approach:

Create separate security group for your lambda
Allow inbound traffic on port 3306 in RDS sg for lambdas sg.

edited May 26 '21 at 09:42

answered Mar 05 '17 at 17:49

Vor

27,623
37
119
184

1

Created a separate security group for the Lambda and have All traffic inbound on the RDS but still the same problem... – Sir Codesalot Mar 06 '17 at 06:10
5

Lifesaver. Who would have thought that AWS would prevent comms within the same security group by default? NONE of the AWS tutorials mention this, they are clear you need your Lambda and RDS in the same group, but fail to mention you'll need to enable them to communicate. (My preferred way is to add an InBound rule to allow all TCP traffic from within the same security group, but the suggestion to create a new one for Lambda and enable it would also work of course.) – hawkip Jul 30 '18 at 19:24

score 14 · Accepted Answer · answered Mar 06 '17 at 06:39

14

I want to thank everyone who helped, the problem turned out to be different than I thought. The callback in the code doesn't work for some reason even though it's in AMAZON'S OWN DEFAULT SAMPLE.

The working code looks like this:

'use strict';
console.log("Loading getContacts function");

var AWS = require('aws-sdk');
var mysql = require('mysql');

exports.handler = (event, context) => {

   var connection = mysql.createConnection({
        host     : '...',
        user     : '...',
        password : '...',
        port     : 3306,
        database: 'ebdb',
        debug    :  false
    });

    connection.connect(function(err) {
      if (err) context.fail();
      else context.succeed('Success');
    });

};

answered Mar 06 '17 at 06:39

Sir Codesalot

4,640
2
33
40

I fought this for over an hour - close to two. I thought my firewall rules were hosed. OMG, how can simply deleting the callback line fix everything? Anyway, good tip, I had done the same thing. Must be some kind of callback deadlock or something. – Quad64Bit Apr 26 '17 at 04:56
1

You need to end the connection before calling the callback. As the connection remains open, the lambda times out. Need to add something like this to the callback of `.connect()`, `connection.end(function (err) { callback(null, response);});`. – fireant May 21 '17 at 08:39
1

Came across this answer - just want to point out that the callback param is optional depending on your NodeJS version, as per the AWS Docs: https://docs.aws.amazon.com/lambda/latest/dg/nodejs-prog-model-handler.html – jarodsmk Feb 14 '19 at 05:24

score 2 · Answer 4 · answered Jan 31 '18 at 10:48

2

When you originally setup the DB, it will automatically create a security group. defaulting to the IP that you set the DB up with. When you run from lambda this rule blocks traffic. Check out your db error logs and you can confirm it is refusing the connection.

***** could not be resolved: Name or service not known

You need to create a rule in the security group to allow lambda traffic. Go to your RDS instance console and click on the security group, select inbound. There you will see the rules. Then make the call to open up to the world, find the AWS lambda IPs or create a VPC.

answered Jan 31 '18 at 10:48

toonsend

919
8
12

your answer solved my problem, do you know how can I find my AWS Lambda IP? @toonsend – AngelSalazar Feb 02 '18 at 18:46
I don't think it works that way. You'll need to set up a VPC. https://docs.aws.amazon.com/lambda/latest/dg/vpc.html – toonsend Feb 04 '18 at 14:44

score 1 · Answer 5 · answered Mar 05 '17 at 07:15

1

I am sharing my experience while connecting RDS.

You need to enable VPC access for the Lambda function, during which you will assign it a Security Group.

Then, within the Security Group assigned to the RDS instance, you will enable access for the Security Group assigned to the Lambda function.

You can get more info here

answered Mar 05 '17 at 07:15

abdulbarik

5,407
3
31
54

Please read the question. Everything you mentioned is covered in the question already. – Mark B Mar 05 '17 at 14:37
1

Lambda and RDS are both on the same (default) VPC – Sir Codesalot Mar 06 '17 at 06:11

score 1 · Answer 6 · answered Dec 24 '17 at 07:04

I have also faced similar timeout scenario. Issue was not doing connection.end() after connection.connect(). Connection.end() should be done before callback.

Working Code:

  var mysql = require('mysql');

    var connection = mysql.createConnection({
        host     : 'host_name',
        user     : 'root',
        password : 'password'
    });


    module.exports.handler = (event, context, callback) => {

// **Connection to database**      
connection.connect(function(err) {
        if (err) {
          console.error('Database connection failed: ' + err.stack);
          return;
        }
        console.log('Connected to database.');
      });

    // **Hit DB Query**
      connection.query("Query", function(err, rows, fields) {
           console.log(rows);
        });


      //**Close Connection**

connection.end(); ***// Missing this section will result in timeout***

    //**Send API Response**
      callback(null, {
              statusCode: '200',
              body: "Success",
              headers: {
                  'Content-Type': 'application/json',
              },
      });

    };

score 1 · Answer 7 · answered Apr 28 '21 at 05:36

1

It took me around 2 days to figure out the exact issue. In my case, both RDS and Lambda function was in Same VPC, Subnet and security group and added the Required Roles but still was getting Socket timeout exception. I was able to solve the issue by changing inbound and outbound rule by following the below link -

https://aws.amazon.com/premiumsupport/knowledge-center/connect-lambda-to-an-rds-instance/

answered Apr 28 '21 at 05:36

user1705352

71
4

Thank god, I've been struggling with this for a week now – fedemengo Apr 28 '21 at 20:52

score 0 · Answer 8 · answered Jun 06 '17 at 07:18

The problem does not originate from the timeout, but from the way you close the connection. Use .destroy() instead if you do not want to wait for the callback that OR use the callback correctly when closing the connection in .end(function(err) { //Now call your callback });

See this thread for a more in depth explanation.

score 0 · Answer 9 · answered Feb 09 '19 at 18:44

the connection.end() should be after callback:

so working code:

    'use strict';
var mysql = require('mysql');

var connection = mysql.createConnection({
    host     : 'xxxxxx.amazonaws.com',
    user     : 'testuser',
    password : 'testPWD',
    port     : 3306,
    database: 'testDB',
    debug    : false        
});

module.exports.handler = (event, context, callback) => {
    // **Connection to database**      
    connection.connect(function(err) {
        if (err) {
            console.error('Database connection failed: ' + err.stack);
            context.fail();
            return;
        }
      else{ 
            console.log('Connected to database.');
        }
    });

    connection.query('show tables from testDB', function (error, results, fields) {
        if (error) {
            console.log("error: connection failed with db!");
            connection.destroy();
            throw error;
        } else {
            // connected!
            console.log("info: connection ok with db!");
            console.log(results);
            context.succeed("done");
            callback(error, results);
        }
    });

    //Send API Response
    callback(null, {
        statusCode: '200',
        body: 'succeed',
        headers: {
          'Content-Type': 'application/json',
        },
    });

    //Close Connection
    connection.end(); // Missing this section will result in timeout***

};

AWS Lambda RDS connection timeout

9 Answers9

Linked