How DuckLake Simplifies Lakehouse Architecture ft. Jordan Tigani & Hannes Mühleisen скачать в хорошем качестве

How DuckLake Simplifies Lakehouse Architecture ft. Jordan Tigani & Hannes Mühleisen

​Join MotherDuck CEO Jordan Tigani and DuckDB's Hannes Mühleisen for an in-depth discussion about DuckLake, the new lakehouse format that's rethinking how we handle metadata and open table formats. ​Discover what led to DuckLake's creation, how it differs from existing solutions, and what it means for the future of data architecture. ☁️🦆 Start using DuckDB in the Cloud for FREE with MotherDuck : https://hubs.la/Q02QnFR40 ➡️ Follow Us LinkedIn:   / motherduck   X/Twitter :   / motherduck   Blog: https://motherduck.com/blog/ 00:00 Intro 00:33 What is DuckLake ? 02:37 Why Open Table Formats Matter 05:36 The Real Pain Point: Updates in Data Lakes 07:43 Storage Cost & Efficiency with Lakehouse Architecture 10:34 Aesthetic Frustrations with Iceberg 14:57 Is Iceberg the New Hadoop? 17:41 Iceberg's Problems Are Conceptual, Not Just Implementation 23:32 What Is DuckLake, Actually? 25:50 DuckLake = SQL Spec + Parquet + Database 29:41 Developer Simplicity: 3-Step DuckLake Setup 33:54 DuckLake vs Hive Metastore 35:40 High Frequency Updates & Snapshots in DuckLake 39:03 DuckLake at Petabyte Scale 42:37 Use Any Metadata Database (BigQuery, Postgres, etc.) 43:54 Early Feedback & Criticism ("Another Standard") 45:46 Vendors & Catalog Lock-in Concerns 47:13 Why So Few Iceberg Implementations Exist 49:04 MotherDuck's Plans for Hosting DuckLake 50:44 Q&A: Will Vendors Adopt DuckLake? 51:49 Why Avro Should Die 54:53 Just Use Parquet™ 55:14 Access Management in DuckLake 57:55 Why REST APIs Are the Wrong Fit 59:59 Closing Thoughts #duckdb #ducklake #iceberg #lakehouse #datalake #warehouse #icebergvsducklake DuckDB creator Hannes Mühleisen and MotherDuck CEO Jordan Tigani dissect the hype and complexity surrounding open table formats. They explore the core pain points driving the adoption of Apache Iceberg and Delta Lake, questioning if these solutions have over-engineered the modern data lakehouse. This discussion provides a critical look at the data engineering landscape, evaluating the real-world trade-offs between different database architectures for managing large-scale tabular data. Hannes offers a sharp critique of Apache Iceberg's design, comparing its trajectory to Hadoop and highlighting fundamental conceptual flaws. The conversation delves into the "aesthetic" problems of Iceberg, such as the cumbersome catalog server, its inefficient use of Avro for metadata, and a complex REST API that departs from its original file-based simplicity. This complexity makes building competent writers incredibly difficult, which explains the ongoing challenges with full DuckDB Iceberg support and the lack of diverse implementations in the ecosystem. Discover DuckLake, a new open table format that radically simplifies the data stack. Born from "spite engineering," DuckLake rejects custom services and instead leverages a standard SQL database for all metadata management. This approach eliminates dozens of technologies, replacing complex catalog servers and APIs with simple SQL queries against a database schema. We explain how this elegant database architecture makes implementing DuckLake straightforward—all you need is the ability to talk to an object store and a database. Learn how DuckLake's design provides significant performance and operational advantages. The format is uniquely suited for high-frequency updates, avoiding the snapshot bloat that plagues Iceberg and enabling thousands of transactions per second. Performance benchmarking on a petabyte of data demonstrates that DuckLake's query planning remains sub-second, addressing common "scaling anxiety." We also showcase how to get a complete DuckLake setup running locally in just three command lines, highlighting its accessibility for developers. Finally, we explore the future of the DuckLake ecosystem, including MotherDuck's plan to offer the best hosted DuckLake experience. The discussion covers practical considerations like access management, per-column encryption, and why its simplicity incentivizes adoption by other vendors. For any data engineer, analyst, or developer building a data lakehouse, this video explains what DuckLake is and why its first-principles approach offers a more scalable, efficient, and elegant alternative to existing open table formats.