12 Joel Roland Steve Lee Solving Data Engineering Challenges скачать в хорошем качестве

12 Joel Roland Steve Lee Solving Data Engineering Challenges

ETL and Data Ingestion Made Easy Organisations have a wealth of information siloed in various data sources. These could vary from databases (Oracle, MySQL, Postgres, etc) to product applications (Salesforce, Marketo, HubSpot, etc). A significant number use cases need data from these diverse data sources to produce meaningful reports and predictions. For many years, organisations tried to centrally collect all their data in the data warehouse but these were not suited or were expensive for handling unstructured data, semi-structured data, and data with high variety, velocity, and volume. It also limited the types of analytics data teams could use; unable to do machine learning or anything more than basic SQL. Delta Lake, released and open-sourced by Databricks in 2019, is helping thousands of organisations build central data repositories in an open format much more reliably and efficiently than before. Delta Lake on Databricks provides Acid transactions and efficient indexing that is critical for exposing the data for various access patterns, ranging from ad-hoc SQL queries in BI tools, to scheduled ML jobs. This session will provide the one-two step of Data Ingestion & Quality made easy with Databricks, which includes Steve Lee, guest presenter from Atlassian, who will talk about how their data team tackles compliance and regulations (GDPR, CCPA) and their approach to self-service data ingestion.