Stack Overflow
Stack Overflow

Questions tagged [apache-airflow]

Airflow is a platform to programmatically author, schedule, and monitor workflows.

220 questions
49
votes
5 answers

How to stop/kill Airflow tasks from the UI

How can I stop/kill a running task on Airflow UI? I am using LocalExecutor. Even if I use CeleryExecutor, how do can I kill/stop the running task?
Chetan J
  • 1,337
  • 4
  • 13
  • 20
29
votes
3 answers

How to add new DAGs to Airflow?

I have defined a DAG in a file called tutorial_2.py (actually a copy of the tutorial.py provided in the airflow tutorial, except with the dag_id changed to tutorial_2). When I look inside my default, unmodified airflow.cfg (located in ~/airflow), I…
Aleksey Bilogur
  • 3,286
  • 2
  • 22
  • 43
28
votes
3 answers

Apache Airflow DAG cannot import local module

I do not seem to understand how to import modules into an apache airflow DAG definition file. I would want to do this to be able to create a library which makes declaring tasks with similar settings less verbose, for instance. Here is the simplest…
fildred13
  • 2,090
  • 6
  • 26
  • 46
26
votes
3 answers

Make custom Airflow macros expand other macros

Is there any way to make a user-defined macro in Airflow which is itself computed from other macros? from airflow import DAG from airflow.operators.bash_operator import BashOperator dag = DAG( 'simple', schedule_interval='0 21 * * *', …
mxxk
  • 6,943
  • 4
  • 30
  • 41
25
votes
2 answers

Airflow structure/organization of Dags and tasks

My questions : What is a good directory structure in order to organize your dags and tasks? (the dags examples show only couple of tasks) I currently have my dags at the root of the dags folder and my tasks in separate directories, not sure is the…
nono
  • 1,685
  • 2
  • 19
  • 28
23
votes
1 answer

Airflow default on_failure_callback

In my DAG file, I have define a on_failure_callback() function to post a Slack in case of failure. It works well if I specify for each operator in my DAG : on_failure_callback=on_failure_callback() Is there a way to automate (via default_args for…
Pierre CORBEL
  • 707
  • 1
  • 6
  • 13
22
votes
1 answer

How to obtain and process mysql records using Airflow?

I need to 1. run a select query on MYSQL DB and fetch the records. 2. Records are processed by python script. I am unsure about the way I should proceed. Is xcom the way to go here? Also, MYSQLOperator only executes the query,…
gpk27
  • 729
  • 1
  • 6
  • 18
22
votes
5 answers

Airflow tasks get stuck at "queued" status and never gets running

I'm using Airflow v1.8.1 and run all components (worker, web, flower, scheduler) on kubernetes & Docker. I use Celery Executor with Redis and my tasks are looks like: (start) -> (do_work_for_product1) ├ -> (do_work_for_product2) ├ ->…
Norio Akagi
  • 545
  • 1
  • 6
  • 17
22
votes
1 answer

How does Airflow's BranchPythonOperator work?

I'm struggling to understand how BranchPythonOperator in Airflow works. I know it's primarily used for branching, but am confused by the documentation as to what to pass into a task and what I need to pass/expect from the task upstream. Given the…
simplycoding
  • 2,256
  • 8
  • 34
  • 70
21
votes
5 answers

Debugging Broken DAGs

When the airflow webserver shows up errors like Broken DAG: [] , how and where can we find the full stacktrace for these exceptions? I tried these locations: /var/log/airflow/webserver -- had no logs in the timeframe of…
arbazkhan002
  • 1,083
  • 2
  • 10
  • 18
18
votes
2 answers

airflow pass parameter from cli

Is there a way to pass a parameter to: airflow trigger_dag dag_name {param} ? I have a script that monitors a directory for files - when a file gets moves into the target directory I want to trigger the dag passing as a parameter the file path.
bsd
  • 947
  • 3
  • 10
  • 25
17
votes
1 answer

(Django) ORM in airflow - is it possible?

How to work with Django models inside Airflow tasks? According to official Airflow documentation, Airflow provides hooks for interaction with databases (like MySqlHook / PostgresHook / etc) that can be later used in Operators for row query…
Pleeea
  • 312
  • 3
  • 11
17
votes
3 answers

Airflow DAG Run triggered, but never executed?

I've found myself in a situation where I manually trigger a DAG Run (via airflow trigger_dag datablocks_dag) run, and the Dag Run shows up in the interface, but it then stays "Running" forever without actually doing anything. When I inspect this DAG…
Aleksey Bilogur
  • 3,286
  • 2
  • 22
  • 43
16
votes
1 answer

Passing parameters to Airflow's jobs through UI

Is it possible to pass parameters to Airflow's jobs through UI? AFAIK, 'params' argument in DAG is defined in python code, therefore it can't be changed at runtime.
Alexander Ershov
  • 833
  • 1
  • 9
  • 22
15
votes
1 answer

How to wait for an asynchronous event in a task of a DAG in a workflow implemented using Airflow?

My workflow implemented using Airflow contains tasks A, B, C, and D. I want the workflow to wait at task C for an event. In Airflow sensors are used to check for some condition by polling for some state, if that condition is true then the next task…
javed
  • 5,106
  • 2
  • 38
  • 63
1
2 3
…
14 15