30

I'm using Celery 2.4.6 and django-celery 2.4.2.

When I configure Celery to use Amazon SQS per the resolution on this question: Celery with Amazon SQS

I don't see anything in the celerycam table in the Django admin. If I switch back to RabbitMQ, the tasks start showing up again.

I have a lot (now 40+) queues in SQS named something like this: "celeryev-92e068c4-9390-4c97-bc1d-13fd6e309e19", which look like they might be related (some of the older ones even have an event in them), but nothing's showing up in the database and I see no errors in the celerycam log.

Any suggestions on what the issue might be or how to debug this further would be much appreciated.

Community
  • 1
  • 1
tobias.mcnulty
  • 1,510
  • 13
  • 14
  • 6
    I haven't found anything conclusive on this, so I ran a few tests to see what backends actually do support celerycam. The key to getting celerycam working is the -E (event sending) to celeryd. In a brief test, it appears that event sending works with RabbitMQ (of course), works with Redis, does NOT work with SQS, and does NOT work with the Django database backend for Celery. – tobias.mcnulty Jan 16 '12 at 19:34
  • 2
    Any further luck with this? I'm attempting to build out a Celery/SQS based system and hitting up against the same problem. – edub Jan 24 '12 at 21:24

1 Answers1

3

SQS is a limited implementation of an AMQP bus. As I understand, it doesn't support PUB/SUB broadcasting like say rabbit-MQ does, which is necessary for events to work properly. SNS was put in place to support broadcasting, but its a separate system.

Some libraries/packages out there are using SimpleDB as a messaging model store as a hack on top of SQS to emulate proper AMQP behavior, but apparently celery does not have a full hack in place yet.

astevanovic
  • 4,246
  • 25
  • 24