51

Is it possible to connect from an AWS Lambda function to a Redis ElastiCache cluster?

I can't figure out if it's a configuration problem or it's simply not possible.

PS: I made a test from an EC2 instance and I can connect to the Redis node. Also the Lambda function and the Redis node are in the same region.

UPDATE (09 Oct 2015):

Amazon announced VPC for AWS Lambda functions. Details here

This means we can now access any resource in AWS behind VPC security group, including ElastiCache and RDS machines.

UPDATE (11 Feb 2016): Amazon launched VPC for AWS Lambda.

https://aws.amazon.com/about-aws/whats-new/2016/02/access-resources-within-a-vpc-using-aws-lambda/

Mircea Soaica
  • 2,689
  • 1
  • 12
  • 25
  • There is no technical restrictions to connect from Lambda to an external service. If you are using third party libraries, you must include them in your ZIP file and these libraries must be able to run on Linux. Can you give a code sample and the error message ? Your question has not enough element included to help you – Sébastien Stormacq Jan 04 '15 at 08:19
  • 1
    I know Lambda can connect to external services. The question is if ElastiCache allows Lambda functions to connect to its clusters. – Mircea Soaica Jan 04 '15 at 10:51
  • 2
    If your security groups are setup correctly, there is no difference from Elastic Cache point of view. This is just a plain TCP connection - there is no difference in connection coming from Lambda or any other service – Sébastien Stormacq Jan 05 '15 at 07:13
  • 2
    From http://docs.aws.amazon.com/AmazonElastiCache/latest/UserGuide/GettingStarted.AuthorizeAccess.html : "All ElastiCache clusters can only be accessed from an Amazon EC2 instance." – Rohit Chatterjee Apr 04 '15 at 17:43
  • 1
    VPC for AWS Lambda announced for "end of 2015" but not released. Now well into 2016 and still no ElasticCache. – David Betts Jan 21 '16 at 17:34
  • Now VPC option is added to lambda, is there any tutorial on how to actually code it to use ElastiCache? – Nick Apr 25 '16 at 10:38
  • Here's a tutorial from AWS for accessing a resource in a VPC (e.g. an ElastiCache cluster or an RDS instance) from within AWS Lambda: http://docs.aws.amazon.com/lambda/latest/dg/vpc-tutorials.html – Aditya Feb 22 '17 at 19:25

5 Answers5

19

As of Feb 2016, AWS allows using lambda functions to connect to Elasticache. Refer to Access Resources within a VPC using AWS Lambda. Here is a link how it works - Tutorial: Configuring a Lambda Function to Access Amazon ElastiCache in an Amazon VPC

ketan vijayvargiya
  • 4,794
  • 1
  • 15
  • 30
greg_diesel
  • 2,770
  • 1
  • 13
  • 23
  • 5
    As of Feb 2016, AWS announced [VPC for Lambda](https://aws.amazon.com/about-aws/whats-new/2016/02/access-resources-within-a-vpc-using-aws-lambda/). – Jordan Arseno Oct 17 '16 at 01:13
  • 7
    Be careful using lambda inside a VPC. It's easy to run into performance issues quickly. We are currently having trouble getting passed just 25 concurrent lambda invocations due to limited ENI capacity, even though we have 4 different subnets provided on the lambda function, with plenty of IP addresses to use. We are currently investigating with AWS support. – JeremyTM Aug 13 '19 at 08:44
4

Setting up an HTTP Proxy or iptables wouldn't work for the following reasons:

Redis calls are not HTTP and will not be handled by HTTP proxies. iptables (or any port forwarding for that matter) will either won't accept a domain name as destination or is highly inefficient due to DNS resolution required every time.

The best and convenient method is to install twemproxy in an EC2 machine and route your requests through it. As a bonus, you suddenly have deployed a fantastic sharding strategy as well.

Nirmal
  • 8,771
  • 10
  • 51
  • 77
  • Exactly what I was looking for, thanks for sharing! We did a similar thing with `pgbouncer` for Amazon RDS and were looking for a solution for Elasticache. Putting our Lambda in a VPC unfortunately causes unacceptable latency for us due to extra startup time from setting up the network interface. – rococo Feb 25 '20 at 07:53
1

I was experiencing the same issue. I did not find a direct solution but instead used a Lambda function to connect to an EC2 server using socket.io which was pretty easy and emit an event to that EC2 server.

When the EC2 server received the event it performed the necessary Redis task ( database cleanup after image thumbnail generation ).

Hope this helps! If anyone finds out how to connect to ElastiCache from Lambda directly I'd still love to know!

R.V.d.M
  • 151
  • 1
  • 5
1

I have tried connecting lambda to memcached elasticache and it works fine. Redis should also be doable.

Couple of things to keep in mind:

  1. Lambda and Elasticache has to be in the same VPC.
  2. When lambda is run in VPC, it won't have access to internet (so access to public APIs won't work). NATGateway is required for this.
Rssr
  • 11
  • 2
0

You can use (Redislabs) managed cloud solution which can be connected without VPC.

There will be no VPC cold starts with cloud solution, but there will be a latency overhead because the call to managed redis is made over internet. The additional latency is around 20 ms when lambda and redis are deployed in the same region. This is manageable for my use case and I don't need to worry about VPC.

Surakshith
  • 156
  • 1
  • 9