4

Anyone know where I can add extra celery configs to airflow celery executor? For instance I want http://docs.celeryproject.org/en/latest/userguide/configuration.html#worker-pool-restarts this property but how do I allow extra celery properties..

Ash Berlin-Taylor
  • 3,335
  • 26
  • 30
Ace Haidrey
  • 928
  • 1
  • 10
  • 26

1 Answers1

7

Use the just-released Airflow 1.9.0 and this is now configurable.

In airflow.cfg there is this line:

# Import path for celery configuration options
celery_config_options = airflow.config_templates.default_celery.DEFAULT_CELERY_CONFIG

which points to a python file from the import path. The current default version can is https://github.com/apache/incubator-airflow/blob/1.9.0/airflow/config_templates/default_celery.py

If you need a setting that isn't tweakable via that file then create a new module, say 'my_celery_config.py':

CELERY_CONFIG = {
    # ....
}

and put it in your AIRFLOW_HOME dir (i.e. along side the dags/ folder) and then set celery_config_options = my_celery_config.CELERY_CONFIG in the config.

Ash Berlin-Taylor
  • 3,335
  • 26
  • 30
  • 2
    expanding on your answer: `from airflow.config_templates.default_celery import DEFAULT_CELERY_CONFIG` `CELERY_CONFIG = dict(DEFAULT_CELERY_CONFIG, **{key1: value1, ...})` will give you all the regular defaults, with your values overriding – Nimrod Morag Dec 11 '18 at 14:10
  • This isn't working for me in the newest version of airflow – Kalimantan Oct 25 '19 at 16:15