6

How to generate tasks dynamically based on the list returned from an upstream task.

I have tried the following:

Using an external file to write and read from the list - this option works but I am looking for a more elegant solution.

Xcom pull inside a subdag factory. It does not work. I am able to pass a list from the upstream task to a subdag but that xcom is only accessible inside of a subdag's task and cannot be used to loop/iterate over the returned list and generate tasks. for e.g. subdag factory method.

 def subdag1(parent_dag_name, child_dag_name, default_args,**kwargs):
    dag_subdag = DAG(
        dag_id='%s.%s' % (parent_dag_name, child_dag_name),
        default_args=default_args,
        schedule_interval="@once",
    )
    list_files='{{ task_instance.xcom_pull( dag_id="qqq",task_ids="push")}}'
    for newtask in list_files:
        BashOperator(
            task_id='%s-task-%s' % (child_dag_name,   'a'),
            default_args=default_args,
            bash_command= 'echo '+ list_files + newtask,
            dag=dag_subdag,
        )
    return dag_subdag
Amit Kumar
  • 405
  • 1
  • 4
  • 17
  • 1
    The `list_files` template will only be evaluated after it is passed as `bash_command` to `BashOperator`. So the for loop will not work as you intended. – eldos Jun 14 '18 at 05:54
  • Agreed with **@eldos**, I'm afraid `Airflow` [doesn't support](https://stackoverflow.com/a/44638526/3679900) generating `DAG` on the fly – y2k-shubham Jul 11 '18 at 12:07
  • 1
    Take a look at this answer, it uses xcoms and subdags to create tasks based on the output of an upstream task: https://stackoverflow.com/a/51977800/9488397 – Christopher Beck Aug 23 '18 at 07:07

0 Answers0