Enterprise Pipeline Orchestration with Cloud Composer and Workflows
Build a production-ready GCP data orchestration layer with Managed Service for Apache Airflow, Cloud Workflows, Dataform, and BigQuery.
Tharun Vempati
·
Oct 11
10 min read
TAG ARCHIVE
Hands-on Lab
4 articles indexed under this topic
Building a Medallion Lakehouse with Dataform (Hands-On Lab)
Build a production BigQuery Medallion Lakehouse with Google Cloud Dataform: incremental SQLX models, late-arrival deduplication, quarantine tables, assertions, and release workflows.
Tharun Vempati
·
Oct 10
15 min read
Building a Production-Ready Real-Time Streaming Pipeline
Build a resilient GCP streaming pipeline with Pub/Sub, Apache Beam on Dataflow, raw-event landing, quarantine handling, event-time windows, and BigQuery Storage Write API sinks.
Tharun Vempati
·
Oct 07
9 min read
Building the Production Storage & Access Layer (Hands-On Lab)
A practical, end-to-end GCP data engineering lab covering Cloud Storage, BigLake external connections, partitioned and clustered BigQuery, materialized views, Time Travel, and authorized views.
Tharun Vempati
·
Sep 30
14 min read