AWS Data Lakes 101 | Lesson 4: Real Time Data Lake Ingest Using Kinesis Firehose скачать в хорошем качестве

AWS Data Lakes 101 | Lesson 4: Real Time Data Lake Ingest Using Kinesis Firehose

☕ https://www.buymeacoffee.com/johnnych... ℹ️ https://johnnychivers.co.uk/ ℹ️ https://github.com/johnny-chivers/dat... ℹ️ https://aws.amazon.com/lake-formation In this series of videos we take a look at AWS Data Lakes using Lake Formation. We mix the theory with the practical as we build an AWS Data Lake using AWS Lake formation, The Glue Data Catalog, Glue Crawlers, Glue ETL, PySpark and Athena. In lesson four we using Kinesis Firehose to ingest real time streaming data into our AWS data lake. We then create a new table using AWS Crawlers and the AWS Glue Data Catalog before accessing the data using AWS Athena. AWS Lake Formation is a service that makes it easy to set up a secure data lake in days. A data lake is a centralized, curated, and secured repository that stores all your data, both in its original form and prepared for analysis. A data lake enables you to break down data silos and combine different types of analytics to gain insights and guide better business decisions. However, setting up and managing data lakes today involves a lot of manual, complicated, and time-consuming tasks. This work includes loading data from diverse sources, monitoring those data flows, setting up partitions, turning on encryption and managing keys, defining transformation jobs and monitoring their operation, re-organizing data into a columnar format, configuring access control settings, deduplicating redundant data, matching linked records, granting access to data sets, and auditing access over time. Creating a data lake with Lake Formation is as simple as defining data sources and what data access and security policies you want to apply. Lake Formation then helps you collect and catalog data from databases and object storage, move the data into your new Amazon S3 data lake, clean and classify your data using machine learning algorithms, and secure access to your sensitive data. Your users can access a centralized data catalog which describes available data sets and their appropriate usage. Your users then leverage these data sets with their choice of analytics and machine learning services, like Amazon Redshift, Amazon Athena, and (in beta) Amazon EMR for Apache Spark. Lake Formation builds on the capabilities available in AWS Glue. 😎 About me I have spent the last decade being immersed in the world of big data working as a consultant for some the globe's biggest companies. My journey into the world of data was not the most conventional. I started my career working as performance analyst in professional sport at the top level's of both rugby and football. I then transitioned into a career in data and computing. This journey culminated in the study of a Masters degree in Software development. Alongside many a professional certification in AWS and MS SQL Server.