7

I have been experimenting with deploying Django apps to AWS Lambda with Zappa.

In some of my other (EC2/EBS hosted) Django projects, if there is a need to perform some heavier calculation that can take some time (such as sending a lot of emails, or just some processing that takes over a minute), Celery is used. It is a task queue system where the tasks are sent to a queue, a response can be returned immediately and workers can process the tasks later.

What would be the best way to implement a Celery-like task queuing system for a Zappa-Django app running in Lambda?

Zappa/Lambda supports scheduled tasks, and the models of the app could be designed in such a way that the processing could be done by scheduled functions later and the results could be saved to DB. But I do not think polling for tasks once a minute is robust enough, there is a oftena need to start the delayed task immediately.

Is there an easy way to return a response from a Django view immediately and have a function (from inside the Django app) with arbitrary parameters queued to be executed later?

tuoppimas
  • 101
  • 1
  • 3

2 Answers2

4

You can do it using SNS. Subscribe lambda to topic and publish messages there with json payload.

Raz
  • 5,732
  • 3
  • 24
  • 23
  • I am not sure this would be on optimal solution. It could work, but would the task consumers have to be separate code running in another lambda function? Also, all data is not JSON serializable and environment variables etc. from the django app would not be available. – tuoppimas Jan 13 '17 at 18:07
  • Celery works in the same way. Task arguments are serialized to json and another process executes tasks. You can include config file to the "worker" lambda if you need variables. – Raz Jan 13 '17 at 20:17
  • Thank you, I will look into it. But it will nevertheless require another Lamba function "deployment"? Maybe two separate django-zappa deployments could be the solution - the other one automatically stripped from all other code except for the tasks. – tuoppimas Jan 13 '17 at 20:29
0

I made a db driven task queue for zappa. https://github.com/andytwoods/zappa-call-later . Early days, but we are using it in production.

Every X minutes, a Zappa event pings a function that checks for tasks. Tasks can be delayed, repeated etc.

andyw
  • 2,309
  • 1
  • 24
  • 38