Stop Treating Cloud Composer (Airflow) Like a Data Processing Engine
One of the most expensive anti-patterns I see in Google Cloud data platforms is teams scaling up Cloud Composer (Apache Airflow) worker nodes because their DAGs keep failing with OOMKilled.
When you inspect the DAG code, the culprit is almost always the same: a PythonOperator pulling 5 million rows into a Pandas DataFrame in memory, transforming it on the Airflow worker, and writing it back out.
Airflow is an air-traffic controller, not a cargo plane.
The control tower’s job is to trigger flights, track dependencies, handle retries, and alert when a runway is blocked—not carry the luggage on its own back.
In a healthy GCP data architecture:
- Cloud Composer / Cloud Workflows sends lightweight API calls and waits on sensors.
- BigQuery + Dataform handles SQL transformations inside Google’s petabyte shuffle engine.
- Dataflow / Cloud Run Jobs handles custom streaming or heavy non-SQL processing.
The moment you remove data payloads from your orchestrator, your Composer cluster shrinks to a fraction of the cost—and your DAG reliability jumps overnight.
Airflow is the air-traffic controller, never the cargo plane—keep orchestrator tasks stateless and push every byte of compute to BigQuery, Dataform, or Dataflow.