Скачать с ютуб Product Management perspective on Data Observability with Databand в хорошем качестве

Product Management perspective on Data Observability with Databand

Presented by IBM at Airflow Summit 2024. Steve Sawyer discusses a case study for how IBM Data Observability with Databand, collects metadata to build historical baselines, detect anomalies and triage alerts to remediate data quality issues for you data pipelines and warehouses. Additionally, he will provide a Product perspective on the technologies IBM is building to meet the data observability needs across the enterprise, and how it relates to our investments in AI and Data Fabric. ----- (AI Generated summary) Key Takeaways: *Data is the new differentiator:* In today's market, unique design and distribution are no longer enough. High-quality data is crucial for building differentiated products, especially in AI, where model performance hinges on data quality. *Data trust is lacking:* CEOs and CIOs express concerns about the reliability of their data, particularly for use in AI models and critical workflows. *Databand addresses data trust issues:* IBM acquired Databand to provide observability into data pipelines, enabling better decision-making and higher-quality products. *Databand focuses on:* *Early issue detection:* Identifying problems early in data pipelines, particularly for ML workflows, prevents downstream performance issues. *Data trust:* Providing tools for faster resolution of data quality issues. *Faster resolution:* Streamlining troubleshooting and directing the right personnel to address issues quickly. *Databand's capabilities:* *Unified view:* Integrates with various data tools (Airflow, Azure Data Factory, DBT, Spark, etc.) for a consolidated view of data pipelines. *Detailed run information:* Provides insights into pipeline runs, including task-level durations, parameters, metrics, and logs from all integrated systems. *Data set monitoring:* Tracks data set schemas, record counts, and data quality metrics, allowing for trend analysis and anomaly detection. *Flexible alerting:* Offers customizable alerts for various pipeline and data quality issues, with integrations for Slack, PagerDuty, email, and more. *Impact analysis:* Identifies potentially affected data sets and downstream pipelines in case of failures, aiding in faster root cause analysis. *Databand streamlines troubleshooting:* Eliminates the need to manually investigate individual systems by providing direct links to relevant data and logs within the context of alerts. *Overall, Databand empowers organizations to build trust in their data, improve data quality, and accelerate product development by providing comprehensive observability into their data pipelines.*