Real-Time Reporting and Analytics for Construction Data Powered by Delta Lake and DBSQL скачать в хорошем качестве

Real-Time Reporting and Analytics for Construction Data Powered by Delta Lake and DBSQL

Procore is a construction project management software that helps construction professionals efficiently manage their projects and collaborate with their teams. Our mission is to connect everyone in construction on a global platform. Procore is the system of record for all construction projects. Our customers need to access the data in near real-time for construction insights. Enhanced reporting is a self-service operational reporting module that allows quick data access with consistency to thousands of tables and reports. Procore data platform rebuilt the module (originally built on the relational database) using Databricks and Delta lake. We used Apache Spark™ streaming to maintain the consistent state on the ingestion side from Kafka and plan to leverage the fully capable functionalities of DBSQL using the serverless SQL warehouse to read the medallion models (built via DBT) in Delta Lake. In addition, the Unity Catalog and the Delta share features helped us share the data across regions seamlessly. This design enabled us to improve the p95 and p99 read time by xx% (which were initially timing out). Attend this session to hear about the learnings and experience of building a Data Lakehouse architecture. Talk by: Jay Yang and Hari Rajaram Connect with us: Website: https://databricks.com Twitter:   / databricks   LinkedIn:   / databricks   Instagram:   / databricksinc   Facebook:   / databricksinc