November Data: Unlocking Observability with ClickHouse Columnar Storage скачать в хорошем качестве

November Data: Unlocking Observability with ClickHouse Columnar Storage

November's data revolution! Discover why ClickHouse is the ultimate solution for observability, offering unparalleled performance for logs, metrics, and traces. See ClickHouse’s advantages, use cases and unlock the power of columnar storage for your data needs. This video explores how ClickHouse, an open-source columnar database, provides a superior alternative to traditional search engines and metric stores for handling large-scale observability data. We dive deep into its architecture, benefits, and practical applications within the Clickstack observability stack. This video will premiere on 2026-02-01 11:00 Europe/Amsterdam. What You'll Learn: • Understand the limitations of search engines and metric stores for observability. • Discover the power of columnar storage and ClickHouse for handling logs, metrics, and traces. • Learn how Clickstack leverages ClickHouse, HyperDX, and OpenTelemetry for a complete observability solution. TIMESTAMPS/CHAPTERS: 0:00 Introduction 2:15 ClickHouse Overview 5:30 Clickstack Explained 8:45 HyperDX and ClickHouse 12:00 Data Store Options 15:45 Search Engine Limitations 19:00 Metric Store Drawbacks 22:30 Key Requirements for a Data Store 26:00 Columnar Storage Explained 30:00 Column Storage Benefits 34:00 Stack Overflow Example 38:00 JSON Support in Column Stores 42:00 S3 Storage Integration 46:00 ClickHouse Advantages 50:00 SQL and Observability 54:00 OpenTelemetry Integration 58:00 UI Design for Optimal Performance 1:02:00 Query Optimization Detailed Topics Covered: We begin by examining the challenges of using traditional search engines like OpenSearch and Elasticsearch for observability, highlighting their limitations in aggregation and high cardinality data. Then, we explore metric stores like Prometheus, noting their constraints in search capabilities and exploratory workflows. Next, we delve into the core principles of columnar storage, explaining how ClickHouse’s architecture enables efficient compression, fast aggregations, and parallel query execution. The video further discusses ClickHouse's support for JSON data, integration with S3 object storage, and schema-on-read/write capabilities. We also explore real-world examples and use cases, demonstrating how ClickHouse can be used to build powerful observability stacks for managing logs, metrics, and traces. Key Insights & Takeaways: ClickHouse offers a cost-effective and scalable solution for storing and analyzing observability data, surpassing the limitations of search engines and metric stores. Columnar storage enables fast aggregations, efficient compression, and parallel query execution. The integration of ClickHouse with tools like HyperDX and OpenTelemetry creates a comprehensive and open-source observability stack. By understanding the principles of columnar storage and its benefits, you can optimize your data infrastructure for improved performance and scalability. Additional Resources: ClickHouse GitHub: HyperDX: OpenTelemetry: About This Channel: Luca Berton is dedicated to providing practical, hands-on tutorials and insightful content on data engineering, cloud computing, and data science. Subscribe for weekly videos exploring cutting-edge technologies and real-world applications, with the goal of helping you master modern data tools. If you found this video helpful, please give it a like, leave a comment with your questions or thoughts, and subscribe to the channel for more content on data engineering and observability! Don't forget to hit the notification bell so you never miss an upload.