Anyone know where I can add extra celery configs to airflow celery executor? For instance I want http://docs.celeryproject.org/en/latest/userguide/configuration.html#worker-pool-restarts this property but how do I allow extra celery properties..
Asked
Active
Viewed 3,363 times
1 Answers
7
Use the just-released Airflow 1.9.0 and this is now configurable.
In airflow.cfg there is this line:
# Import path for celery configuration options
celery_config_options = airflow.config_templates.default_celery.DEFAULT_CELERY_CONFIG
which points to a python file from the import path. The current default version can is https://github.com/apache/incubator-airflow/blob/1.9.0/airflow/config_templates/default_celery.py
If you need a setting that isn't tweakable via that file then create a new module, say 'my_celery_config.py':
CELERY_CONFIG = {
# ....
}
and put it in your AIRFLOW_HOME dir (i.e. along side the dags/ folder) and then set celery_config_options = my_celery_config.CELERY_CONFIG
in the config.
![](../../users/profiles/439189.webp)
Ash Berlin-Taylor
- 3,335
- 26
- 30
-
2expanding on your answer: `from airflow.config_templates.default_celery import DEFAULT_CELERY_CONFIG` `CELERY_CONFIG = dict(DEFAULT_CELERY_CONFIG, **{key1: value1, ...})` will give you all the regular defaults, with your values overriding – Nimrod Morag Dec 11 '18 at 14:10
-
This isn't working for me in the newest version of airflow – Kalimantan Oct 25 '19 at 16:15