0

The context here is simple, there's a lambda (lambda1) that creates a file asynchronously and then uploads it to S3.

Then, another lambda (lambda2) receives the soon-to-exist file name and needs to keep checking S3 until the file exists.

I don't think S3 triggers will work because lambda2 is invoked by a client request

1) Do I get charged for this kind of request between lambda and S3? I will be polling it until the object exists

2) What other way could I achieve this that doesn't incur charges?

3) What method do I use to check if a file exists in S3? (just try to get it and check status code?)

Mojimi
  • 1,651
  • 5
  • 30
  • 81
  • Did you use trigger between lambda and S3? – Vihar Manchala Mar 06 '19 at 20:59
  • @ViharManchala wouldn't work as lambda2 is invoked by a client – Mojimi Mar 06 '19 at 21:01
  • You will be billed for the lambda execution time. I would recommend you to use trigger. – Vihar Manchala Mar 06 '19 at 21:09
  • Could you please Edit your question and explain more about the scenario? For example, what do you mean by "lambda1 creates a file asynchronously"? Does the Lambda function wait for the fail to be created or does it exit before the file is created? (Then how is the file actually created?) What do you mean by "lambda2 needs to keep checking" - do you mean that lambda2 executes once and stays in a loop until it seems the file? How long does it typically wait, given that there is a 15-minute time limit on a Lambda function? What do you mean by "lambda2 is invoked by a client request"? – John Rotenstein Mar 06 '19 at 21:46

2 Answers2

1

This looks like you should be using an S3 objectCreated trigger on the Lambda. That way, whenever an object gets created, it will trigger your Lambda function automatically with the file metadata.

See here for information on configuring an S3 event trigger

Deiv
  • 2,361
  • 2
  • 14
  • 25
  • 1
    Edited my question, I don't think this works as it's invoked via client request – Mojimi Mar 06 '19 at 21:02
  • I see, in that case, you can see [this post for how to check if the object exists](https://stackoverflow.com/q/33842944/7207514) – Deiv Mar 06 '19 at 21:04
0

Let me make sure I understand correctly.

  1. Client calls Lambda1. Lambda1 creates a file async and uploads to S3 the call to lambda one returns as soon as lambda1 has started it's async processing.
  2. Client calls lambda2, to pull the file from s3 that lambda1 is going to push there.

Why not just wait for Lambda one to create the file and return it to client? Otherwise this is going to be an expensive file exchange.

Neil
  • 549
  • 3
  • 13
  • Well, the client might be waiting several minutes for a response, I don't think that's good design – Mojimi Mar 06 '19 at 21:36
  • I was thinking of playing around the fact that delete requests are free, so I would do a dummy file along the main file and keep trying to delete it, when it suceeds I know the main file is ready :D – Mojimi Mar 06 '19 at 21:37
  • Isn’t the client going to be waiting with your approach as well? – Neil Mar 06 '19 at 21:40
  • He's waiting for the file but not the response, since he's constantly getting "file not ready yet" responses, this is to contour timeouts – Mojimi Mar 06 '19 at 21:44
  • So the question is how does the client get notified that the file is ready. – Neil Mar 06 '19 at 22:09
  • @Mojimi note that in S3, delete requests always succeed, even on nonexistent objects. – Michael - sqlbot Mar 07 '19 at 02:13