1

Is there any AWS managed solution which would allow be to perform what is essentially a data migration using DynamoDB as the source and a Lambda function as the sink?

I’m setting up a Lambda to process DynamoDB streams, and I’d like to be able to use that same Lambda to process all the existing items as well rather than having to rewrite the same logic in a Spark or Hive job for AWS Glue, Data Pipeline, or Batch. (I’m okay with the input to the Lambda being different than a DynamoDB stream record—I can handle that in my Lambda—I’m just trying to avoid re-implementing my business logic elsewhere.)

I know that I could build my own setup to run a full table scan, but I’m also trying to avoid any undifferentiated heavy lifting.

Edit: One possibility is to update all of the items in DynamoDB so that it triggers a DynamoDB Stream event. However, my question still remains—is there an AWS managed service that can do this for me?

Matthew Pope
  • 5,551
  • 1
  • 19
  • 36
  • 1
    how about doing minor modifications in all the dynamodb entries(probably adding a random field), so that they go into dynamodb streams. – best wishes Apr 06 '19 at 01:47
  • That was one of the things I was thinking. I would like, as much as possible, to use managed AWS services since this is a one time thing, and I don't want to have to build my own fault tolerance for a one-time table scan job. – Matthew Pope Apr 06 '19 at 04:46

1 Answers1

0

You can create a new kinesis data stream. Add this as a trigger to your existing lambda function. Create a new simple lambda function which scans the entire table and puts records into this stream. That's it.

Your business logic stays in your original function. You are sending existing data from dynamodb to this function via kinesis.

Ref: https://aws.amazon.com/blogs/compute/indexing-amazon-dynamodb-content-with-amazon-elasticsearch-service-using-aws-lambda/

Sasank Mukkamala
  • 1,246
  • 11
  • 22