2

So I have a very confusing issue that I don't know how to solve. My setup is API Gateway -> Lambda -> IoT Core. I setup the code and it works fine from my IDE. I deploy it to AWS, and my connection to AWS times out.

The Lambda is in a single subnet and the subnet does have a default route to an IGW. I did a test, and the Lambda function can resolve the IP of my IoT endpoint to a public IP (54.x.x.x). But the connect() method times out. My security group for the Lambda function is setup to allow all incoming / outgoing.

What am I missing? Why can't I get to IoT Core from inside a VPC with an IGW configured and seems to be working. Any direction would be greatly appreciated.

UPDATE

After playing around with many different things, I can't identify what exactly I had messed up with my configuration. But following the accepted answer I ended up with the following setup which appears to work for what I need.

  • subnet-1 10.14.10.0/24 (auto-assign-public=false)
    • local route ( 10.14.0.0/16 ) and default route=nat-gateway
  • subnet-2 10.14.20.0/24 (auto-assign-public=false)
    • local route ( 10.14.0.0/16 ) and default route=nat-gateway
  • subnet-3 10.14.30.0/24 (auto-assign-public=false)
    • local route ( 10.14.0.0/16 ) and default route=nat-gateway
  • subnet-4 10.14.40.0/24 (auto-assign-public=false)
    • local route ( 10.14.0.0/16 ) and default route=nat-gateway
  • subnet-5 10.14.200.0/24 (auto-assign-public=true)
    • local route ( 10.14.0.0/16 ) and default route=igw
  • nat-gateway
    • in subnet-5

I don't know if this is what I intended, but this is what I was looking for. A series of subnets that are not publicly accessible, but has an internet connection for access to other AWS services. So my Lambda resources, ECS, etc can sit privately and access what they need.

Thank you everyone for the information.

cinnafire
  • 23
  • 3
  • Does all work when lambda is not in VPC? Does your lambda has correct execution role for VPC? – Marcin Mar 24 '20 at 23:17
  • Yes, API GW has multiple endpoints all of which work fine inside the VPC (accessing DynamoDB, elasticache, etc). I haven't specifically tried outside of VPC. I'll try that for a test. – cinnafire Mar 24 '20 at 23:45
  • @Marcin yes, it does work outside of the VPC. – cinnafire Mar 25 '20 at 00:02
  • Interesting. Do you have any NACLs on your subnet that can block connections. Also its your subnet private or public? – Marcin Mar 25 '20 at 00:25
  • The VPC and subnet are private. I am using the default NACL which has a single rule of allow all ( 100 / ALL Traffic / ALL / ALL / 0.0.0.0/0 / ALLOW ) for both inbound and outbound. I also have a NAT GW in this same subnet which I just noticed. I added that to fix something in the past but I don't remember why. – cinnafire Mar 25 '20 at 00:29
  • So you have NAT gateway in private subnet? NAT gateway should be in public subnet. – Marcin Mar 25 '20 at 00:34
  • I did, but just removed it. Its listed as "deleted" now and my API still appears to work in other aspects, but still broken in that I can't access AWS IoT Core. My "main route table" for the VPC shows the default route to be my IGW. The subnet in question also lists that same IGW for the default route. Is that the problem? Do I need to make the default route for my subnet point to something else? I'm at a total loss. – cinnafire Mar 25 '20 at 01:13

2 Answers2

2

You should not deploy the Lambda function to a public subnet (that's the subnet with the default route to the IGW). It won't work the way you want it to work. The Lambda function doesn't have, and cannot have, a public IP so cannot route to the internet via the IGW.

If the Lambda needs to be in VPC, then move it to a private subnet and make sure that the private subnet has a default route to a NAT (or NAT gateway) in a public subnet. Or deploy the Lambda function outside of VPC completely, if that's viable.

More information at:

jarmod
  • 46,751
  • 9
  • 81
  • 86
  • All my "private" subnets have a default route to the IGW... but I just assumed the IGW acted as a router to route my 10.x.x.x IP traffic to a public source if needed through the IGW. I thought what made them private was the fact they didn't have a public IP. Although that does sound like a NAT function, so maybe I need to read a little more on both. Should my private subnets (sub-a, sub-b, etc) have a default route to my public subnet (sub-z) which would be the only one with the default route to the IGW ? Maybe I thought I had a working internet connection in my VPC when I didnt. – cinnafire Mar 25 '20 at 01:47
  • 1
    No, your private subnets don't have a default route to an IGW. Those are public subnets. All subnets have a local target (if it's a 10/16 network then the destination is 10.0.0.0/16 and the target is local) - this is how VPC subnets communicate with each other. Public subnets have a default route to an IGW. Private subnets can have a default route to a NAT in a public subnet. See https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Scenario2.html for more. – jarmod Mar 25 '20 at 01:52
  • Thanks for the link. I'll read and try to get it working, then update my original question with my resolution (if I am able to figure it out haha). Thanks for the info. Cloud networking is a "fun" – cinnafire Mar 25 '20 at 01:59
  • 2
    Move the Lambda to a private subnet, ensure you have a NAT in a public subnet, and add a default route from the private subnet to that NAT. It should be straightforward to set up. – jarmod Mar 25 '20 at 02:06
0

When you say "I did a test, and the Lambda function can resolve the IP of my IoT endpoint to a public IP (54.x.x.x)" Do you mean DNS resolution, or you've checked this with a actual network traffic.

In either case, you can turn VPC Flow Logs for your VPC, and try again. The flow log will identify whether SGs or NACLs are blocking your traffic.

Remember also that Lambda's cannot exists in a public subnet, they have to reside in private subnets, and use NAT GW on public subnets to connect to the internet.

keithRozario
  • 316
  • 1
  • 4
  • I mean DNS resolution, which I guess could have been done with an "internal" DNS server inside my VPC if AWS provides such a thing. I guess I don't really understand when to use an IGW and a NAT GW. I have an IGW now, but why wouldn't that permit internet on my VPC if that is the default route? I'll look into VPC flow logs. I've seen them before, just never used them. – cinnafire Mar 25 '20 at 01:18
  • VPC does include DNS resolution for local domain names for the EC2 instances (like ec2-192-0-1-2.compute-1.amazonaws.com) and it performs recursive lookups against public name servers. – jarmod Mar 25 '20 at 01:21