4

I have a large file being uploaded on S3, and for each line in the file I need to make a long running rest API call. I'm trying to figure out the best way to break up the work. My current flow idea is

Lambda (break up file by line) -> SNS (notification per line) -> Lambda (separate per line/notification)

This seems like it is a common use case, but I can't find many references to it, am I missing something? Is there a better option to break up my work and get it done in a reasonable amount of time?

thefroatgt
  • 856
  • 1
  • 12
  • 18
  • 2
    Why not skip the middle step and just call a second lambda function from the first one? Kind of like this: http://stackoverflow.com/questions/31714788/can-an-aws-lambda-function-call-another – JohnAllen Dec 29 '15 at 23:47
  • How long is your `long running rest API call`? – helloV Dec 31 '15 at 02:00

1 Answers1

2

The Best way is going to be subjective. The method you are using currently, Lambda->SNS->Lambda, is one possible method. As JohnAllen pointed out, you could simply do Lambda->Lambda.

Your scenario reminds me of this project, which has a single Lambda function adding items to a Kinesis stream, which then triggers many parallel Lambda functions.

I think Lambda->Kinesis->Lambda might be a better fit for your use case than Lambda->SNS->Lambda if you are generating a very large number of Lambda tasks. I would be worried that the SNS implementation would run up against the maximum number of concurrent Lambda functions, while the Kinesis implementation would queue them up and handle that gracefully.

Mark B
  • 139,343
  • 19
  • 240
  • 237