Weeks 9–12. Your BrightCart pipeline gets a schedule (Airflow), a way to handle data too large for one laptop (Spark), a way to ship itself anywhere (Docker) — and then you build something entirely your own for the capstone.
Confirm these before Week 9 — Phase 3 assumes all of it is second nature.
Before starting Phase 3, confirm you can:
Everything in this phase is really one idea, applied three ways: a pipeline isn't finished when it works once. It has to rerun safely, log enough to debug itself, fail loudly on bad data, and be easy for someone else to deploy and understand. Airflow, Spark, and Docker are three different answers to "how do I make that actually true?"
Phase 2's pipeline runs by hand. Airflow is what runs it for you, on a schedule, with retries, while you sleep.
A single pipeline script is fine until you have five of them, some depending on others finishing first, all needing to run at specific times, with someone needing to know immediately when one fails. Orchestration is the layer that manages all of that — scheduling, dependencies, retries, and visibility.
from airflow import DAG
from airflow.operators.empty import EmptyOperator
from datetime import datetime
with DAG(
dag_id="first_brightcart_pipeline",
start_date=datetime(2026, 1, 1),
schedule="@daily",
catchup=False
) as dag:
start = EmptyOperator(task_id="start")
finish = EmptyOperator(task_id="finish")
start >> finish| Term | Meaning |
|---|---|
| DAG | a workflow definition — a set of tasks and the order they run in |
| Task | one unit of work inside a DAG |
| Operator | a template for a kind of work (run Python, run Bash, etc.) |
Chapter 1 Checklist — before moving on, you should be able to:
PythonOperator & BashOperator, the TaskFlow API & XCom, Variables & Connections, retries, idempotent output paths, and a full realistic BrightCart DAG you can deploy.
PySpark DataFrames, explicit schemas, groupBy/joins, broadcast joins, lazy evaluation, partitioned Parquet output, and launching jobs with spark-submit.
Dockerfiles, volumes, Docker Compose running your pipeline alongside Postgres, and the image-hygiene checklist that keeps builds portable.
Design your own end-to-end data platform on a dataset you choose — ingest, process, store, visualize — plus repo structure, README template, and a 4-day working plan.
Get the complete orchestration, big data, containerization, and capstone chapters — with full code, diagrams, and the README/repo templates you'll present with.