8

Hope someone can help enlighten me on this issue. I am currently working on a lambda function that utilizes the cloud watch scheduler to check various devices and it is using elasticache to maintain a simple database on the readings.

My problem is that after I shut down my testing at night. I fire up the lambda function in the morning and the function has lost access to the internet. Which is represented by the function timing out. Regularly after a few hour of messing around with my routes and my vpc settings it will start working again. Just to break the following day. Sometimes it works with nat gateway other times with just a nat instance. The changes I typically make to the vpc set up are minor. The pattern for the set up I use is one public and one private and one natgateway.

Update: After not being able to access the internet from my VPC all day yesterday, today is functioning fine. What did I do differently, nothing. When it stops functioning again, probably later today, I will be calling up AWS to see if we can get to the bottom of this.

Paul Siskind
  • 91
  • 3
  • 1
    I am having the exact same issue. Were you able to fix it? – skbrhmn Apr 05 '17 at 19:50
  • Did you get to the bottom of it??? – Dalton Sweeney Apr 11 '17 at 15:02
  • It's happening to me as well, I read couple of articles about using a nat gateway and so on... then I configured it, tested it and randomly it loses internet access and sometimes it works just fine D: – mithril_knight Apr 28 '17 at 19:59
  • I've been through the same issue (Lambda function in private subnet, NAT gateway in public subnet, outbound traffic from private subnets routed through NAT). Haven't solved it yet, but I've found that somehow, the function works only the first time it fires in a specific subnet. After that, it fails to reach the internet. – tetele Oct 23 '17 at 16:30
  • 2
    Check out [this question](https://stackoverflow.com/questions/46895254/vpc-running-aws-lambda-sends-sqs-message-only-once). If you only allow a set of ephemeral ports, then "the internet" (i.e. some specific server you are querying) might answer back on some of those ports that you are blocking. – tetele Oct 24 '17 at 07:54
  • when this happened to me, the issue was one of the subnets that the lambda is configured to use, did not have routes to nat gateway – Arun K Dec 04 '19 at 03:29

3 Answers3

3

I've just fixed the same issue with my lambdas - the issue was that I had set the lambda to run in all of my subnets (I have 2 private and 1 public). This knowledgebase article specifies you should run them in private subnets only, which makes sense:

https://aws.amazon.com/premiumsupport/knowledge-center/internet-access-lambda-function/

Go to your lambda page on the AWS console, deselect the public subnet and save and the problem should be solved.

bottleboot
  • 1,631
  • 2
  • 24
  • 39
Liam
  • 3,827
  • 21
  • 32
  • This saved my day! Just one question - should this be strategy for all lambdas (with and without internet access)? – Nirav Bhatt Mar 01 '21 at 18:37
  • 1
    I would suggest it's good practice that you use it for all Lambdas - you may use other AWS services that should potentially only be accessible from your private subnets. – Liam Mar 02 '21 at 07:35
1

It sounds like it is due to the ephemeral port range that AWS Lambda uses. I recommend you check all Network ACLS (NACLS) to ensure that they allow communication on the ephemeral port range used by Lambda:

AWS Lambda functions use ports 1024-65535

So this means that when your lambda runs, it may use any port in this range to send communication to the internet. Even though the destination is port 80 or 443, the sending port will be in this ephemeral range, so when the internet server responds it will send the response back to the originating ephemeral port. Ensure your NACLS allow the communication for this ephemeral range (inbound or outbound or both depending on your use case) or you might be blocked depending on which ephemeral port is used. This article has a useful explanation: https://www.whizlabs.com/blog/ephemeral-ports/

Shawn
  • 5,797
  • 3
  • 23
  • 45
0

A Lambda function with VPC access will require a NAT gateway to access the internet. You state that it sometimes works with only an Internet Gateway, but that isn't possible according to the AWS documentation. If you are removing the NAT gateway, or the VPC's route to the NAT gateway, then that would remove internet access from any Lambda functions that have VPC access enabled.

Mark B
  • 139,343
  • 19
  • 240
  • 237
  • K, maybe it only works with the nat gateway. However, that does not discount the fact that the next day the nat gateway has stopped connecting to the internet without any changes to the configuration. – Paul Siskind Apr 12 '16 at 18:50
  • That sounds like a problem with your NAT gateway instead of the Lambda function. Are you using the managed NAT gateway service or creating your own NAT gateway? – Mark B Apr 12 '16 at 19:48
  • 1
    Guess, I should he more clear in my description. The Lambda function is fine. It's the Nat gateway that is the problem. Normally I select create new nat gateway, spin up a elastic ip and then associate it with my routes at 0.0.0.0/0. – Paul Siskind Apr 12 '16 at 19:59
  • I'm facing same issue where lambda function, under VPC, with all proper configure to access internet, sometime fails to connect with outer internet. – Nilay Anand Jun 25 '19 at 07:25
  • It is possible and is because the lambda is not configured properly - see my answer for how to fix it! – Liam Nov 10 '20 at 12:11
  • 1
    @Liam No it is not possible for a Lambda function running in a VPC to access the Internet with only an Internet Gateway. It requires a NAT Gateway. Your answer specifies that you took your Lambdas out of your public subnets (subnets with only an internet gateway) and ran them only in your private subnets (subnets with a route to a NAT Gateway). So your answer is actually the same as mine. – Mark B Nov 10 '20 at 13:18
  • @MarkB - you are absolutely correct - I had misread the question. – Liam Nov 11 '20 at 12:09