Weeks 5–8. You already know Python and SQL from Phase 1 — now they stop being separate skills and become one automated system: pandas, ETL pipelines, cloud storage, and data warehousing, all built around the same BrightCart dataset.
Two things worth re-grounding before Week 5 begins.
Phase 1 ended with a working brightcart Postgres database, a GitHub repo with real projects, and a star schema you queried by hand. If any of that felt unfamiliar, revisit Phase 1 first — this book assumes it's solid.
pandas
A DataFrame is just a smarter list of dictionaries.
ETL pipelines
Extract/transform/load as code that runs the same way every time.
Cloud & warehousing
Moves BrightCart's data off your laptop and into infrastructure built for scale.
Phase 1 taught you to read one CSV with a for loop. Pandas makes that effortless at real scale — and it's how BrightCart's messiest export finally gets fixed.
Tool Spotlight
A Python library that turns tables of data into objects you can filter, clean, and summarize in a few lines.
What it is
pandas gives you two core objects: a Series (one labeled column) and a DataFrame (a full table). Everything from here on builds on those two ideas.
Why data engineers use it
Everything you did with csv.DictReader and manual for-loops in Phase 1 — filtering, grouping, joining — pandas does in one readable line.
How to get it running
pip install pandas inside an active virtual environment (python -m venv .venv, then activate it). Confirm with python -c "import pandas as pd; print(pd.__version__)".
Time to clean the file that's been waiting since Phase 1's Introduction. It's a denormalized export of BrightCart orders with every problem real data actually has: inconsistent casing, mixed date formats, missing amounts, and duplicate rows.
df = pd.read_csv("data/dirty_orders.csv")
print(df.shape) # (47, 6) <- should be 45 real orders...
print(df.isna().sum())df["customer_name"] = df["customer_name"].str.strip().str.title()
df["status"] = df["status"].str.strip().str.lower()
df["order_date"] = pd.to_datetime(df["order_date"], format="mixed", errors="coerce")
df["amount"] = df["amount"].fillna(0)
df = df.drop_duplicates(subset=["order_id"], keep="first")
print(df.shape) # (45, 6) <- matches the real order count`&` and `|`, not `and` / `or`
& for "and" and | for "or" — Python's own and/or don't work row-by-row across a whole column the way these do. This trips up almost everyone the first time.Chapter 1 Checklist — before moving on, you should be able to:
Series and a DataFrame.shape, .info(), and .isna().sum()Extract, Transform, and Load as three testable functions — plus secrets management, SQLAlchemy loads, retries, idempotency, and validation asserts.
Object storage, IAM & least privilege, serverless compute, and the cost guardrails that keep your first cloud bill from surprising you.
Columnar storage, raw → staging → mart layering, physical star schema tables (CTAS), partitioning/clustering, and query cost mindset.
Your full Phase 2 portfolio evidence checklist, and what changes going into Phase 3 (Airflow, Spark, Docker).
Get the complete ETL pipelines, cloud engineering, and data warehousing chapters — with full code, diagrams, and portfolio project instructions — plus every future update to this book.