How to use xCom in airflow dag file using python operator?

Question

I'm working on this airflow dag file to do some test with XCOM, but not sure how to use it between python operators. Can someone please help how to write the logic to pass a message between the python operators using XCOM push and pull functions. Below is the dag file that i'm working on...

The question is how to pass a message from each task to another task

@dag_factory
def create_dag():
    with DAG(
        dag_id="DSStest",
        default_args=default_args,
        schedule_interval=timedelta(1),
    ) as dag:
        # Define operators here, for example:

        output_file = path_in_workspace("testout")

        rscript_file = path_in_workspace("rtest2.R ")

        bcmd1 = "downloading some file here..."

        t0 = PythonOperator(
            task_id="start",
            python_callable=my_func2,
            provide_context=True,
            op_args=[output_file, 0],
        )

        t1 = PythonOperator(
            task_id="job1",
            python_callable=my_func1,
            provide_context=True,
            op_args=[output_file, 1],
        )

        t2 = PythonOperator(
            task_id="job2",
            python_callable=my_func1,
            provide_context=True,
            op_args=[output_file, 2],
        )

        t10 = PythonOperator(
            task_id="job10",
            python_callable=my_func2,
            provide_context=True,
            op_args=[output_file, 10],
        )

        t20 = BashOperator(
            task_id="job20",
            bash_command=bcmd1,
            queue={
                "worker_type": "png-bash-worker",
                "request_memory": "1G",
                "request_cpu": 1,
            },
        )

        # Define dependencies between operators here, for example:

        t0 >> t1
        t0 >> t2
        t1 >> t10
        t2 >> t10
        t10 >> t11
        t11 >> t20

        return dag  # Do not change this

I think it's not relevant to this example, but is the `@dag_factory` decorator something you've created? I haven't seen that before. — Taylor Edmiston, Aug 16 '18 at 18:55
Possible duplicate of [Python Airflow - Return result from PythonOperator](https://stackoverflow.com/questions/50149085/python-airflow-return-result-from-pythonoperator) — tobi6, Aug 17 '18 at 06:47
Please also see this answer on this question: https://stackoverflow.com/questions/50149085/python-airflow-return-result-from-pythonoperator/50149522#50149522 — tobi6, Aug 17 '18 at 06:47

Christopher Beck · Answer 1 · 2018-08-16T20:37:04.460

3

I recommend to take a look at this example, it shows/explains everything regarding xcoms and PythonOperators:
example_xcom.py

The official airflow page also explains xcoms in a lot of detail:
official documentation

edited Aug 16 '18 at 20:37

answered Aug 16 '18 at 20:25

Christopher Beck

575
6
18

1

That example is great, but let's not pretend the official page explains it in much detail at all, at least at the time of this comment. If you don't already know how xcoms work or have an example to go off of, the existing documentation for this is next to useless, and lacking in necessary context. – Brendan Dec 28 '20 at 22:51

How to use xCom in airflow dag file using python operator?

1 Answers1