From Hot Mess to Happily Ever After: A dbt Glow-Up | Data Valentine Challenge Day 4 скачать в хорошем качестве

From Hot Mess to Happily Ever After: A dbt Glow-Up | Data Valentine Challenge Day 4

Your dbt project looks fine until someone walks the lineage and starts asking questions. In this session, Chloe from Database Tycoon deleted duplicate models, orphan intermediates, unused dimensions, and a Cartesian cross join from Stephen's NYC transit analytics project, live. Stephen volunteered his project for a public review during the Data Valentine Challenge. The format: pull up dbt docs, trace every model in the DAG, and cut anything that doesn't earn its place. During the session, they deleted: Two staging models pulling from the same source (GTFS routes vs. MTA bus routes). One did real work. The other was a bare select. Consolidated to one. Three intermediate models with zero downstream dependencies. Built "just in case." Nothing consumed them. An intermediate model called "stops with routes," a cross join creating a Cartesian product of every stop with every route. Nothing used it. ("You're not gonna need it.") Three unused dimension tables (dim_date, dim_borough, dim_day_type) that nothing joins to. A speculative MetricFlow setup that had no consumers yet. Chloe's mom's rule: when you're getting dressed, always take one accessory off. Then take another one off. That was the whole session in one line. After cleanup, the DAG went from overwhelming to readable, and the remaining problems became obvious. What you'll learn: Why "just in case" models are the #1 source of DAG bloat How to use `dbt docs serve` to audit your lineage in minutes The one rule for deciding what to keep vs. delete --- Recce: https://reccehq.com/ Database Tycoon: https://www.databasetycoon.com/ Data Valentine Challenge: https://reccehq.com/data-valentine-we... Subscribe for more data engineering content.