Xcom In Airflow Here

Here’s a structured, useful blog post about — written for data engineers who want to move beyond basic tasks and build real DAGs. Mastering XComs in Apache Airflow: Cross‑Task Communication Without the Pain One of the first surprises when learning Airflow is that tasks run isolated from each other. You can’t just set task_2.data = task_1.data . So how do you pass a value from one task to another? XComs .

✅ or ensure upstream dependencies with >> . ❌ Using XComs for many small values across many tasks Each XCom is a DB row. 10 000 tasks × 5 XComs = 50 000 rows – fine. But 100 000 tasks × 10 XComs = 1 million rows – slow. Advanced: XCom Backends Airflow 2.0+ lets you store XComs outside the metadata DB. Useful if you need slightly larger values or lower DB load.

process(extract()) # XCom passed implicitly xcom in airflow

Now go build DAGs that actually share information – cleanly and reliably.

process_record(get_latest_record_id()) @task def produce_data(): return "ids": [1,2,3], "source": "api" @task def consume_one(data): return f"Got data['ids'][0]" Here’s a structured, useful blog post about —

XCom (short for cross‑communication ) is Airflow’s built‑in mechanism for exchanging small pieces of data between tasks. When used wisely, they unlock powerful patterns. When abused, they break your DAGs. Let’s see how to use them correctly. XComs are key‑value pairs stored in Airflow’s metadata database. A task can push an XCom (write a value under a key), and another task can pull that value (read it).

from airflow.decorators import dag, task from datetime import datetime @dag(start_date=datetime(2024,1,1), schedule=None, catchup=False) def xcom_demo(): So how do you pass a value from one task to another

push = PythonOperator(task_id='push_task', python_callable=push_function) pull = PythonOperator(task_id='pull_task', python_callable=pull_function)