How to add new DAGs to Airflow?

Question

I have defined a DAG in a file called tutorial_2.py (actually a copy of the tutorial.py provided in the airflow tutorial, except with the dag_id changed to tutorial_2).

When I look inside my default, unmodified airflow.cfg (located in ~/airflow), I see that dags_folder is set to /home/alex/airflow/dags.

I do cd /home/alex/airflow; mkdir dags; cd dags; cp [...]/tutorial_2.py tutorial_2.py. Now I have a dags folder matching the path set in airflow.cfg , containing the tutorial_2.py file I created earlier.

However, when I run airflow list_dags, I only get the names corresponding with the default, tutorial DAGs.

I would like to have tutorial_2 show up in my DAG list, so that I can begin interacting with. Neither python tutorial_2.py nor airflow resetdb have caused it to appear in the list.

How do I remedy this?

score 14 · Accepted Answer · answered Jul 01 '17 at 00:40

14

I think the reason for this is because you haven't exported AIRFLOW_HOME. Try doing: AIRFLOW_HOME="/home/alex/airflow/dags" airflow list_dags. If that's not working than do two steps export AIRFLOW_HOME="/home/alex/airflow/dags" airflow list_dags I believe this should work. Give it a go?

answered Jul 01 '17 at 00:40

Ace Haidrey

928
1
10
26

3

This ought to be the right answer, with the caveat that you *may* need to check and correct the `airflow_home` and `dags_folder` variables in `airflow.cfg` as well. I am still unclear on why three different variables are necessary. cf. [here](https://gitter.im/apache/incubator-airflow?at=59581e6c6ae41d5d33a61aae). – Aleksey Bilogur Jul 01 '17 at 22:20

score 6 · Answer 2 · edited Feb 19 '19 at 18:28

6

In my understanding, AIRFLOW_HOME should link to the directory where airflow.cfg is stored. Then, airflow.cfg can apply and set the dag directory to the value you put in it.

The important point is : airflow.cfg is useless if your AIRFLOW_HOME is not set

edited Feb 19 '19 at 18:28

tatlar

2,502
2
27
32

answered Jul 03 '17 at 06:24

Pierre CORBEL

707
1
6
13

Exactly what Pierre said.^ – Ace Haidrey Jul 03 '17 at 18:43

score 1 · Answer 3 · answered May 02 '19 at 17:16

1

The issue is that you might have two airflow configs existing in your directories, so check for /root/airflow/dags and if yes you require to change the dags_folder path in both airflow.cfg s

answered May 02 '19 at 17:16

ahsen_mukhtar_maken

11
1

How to add new DAGs to Airflow?

3 Answers3