Скачать с ютуб Using Fabric notebooks (pySpark) to clean and transform real-world JSON data в хорошем качестве

Using Fabric notebooks (pySpark) to clean and transform real-world JSON data

10+ hours of FREE Fabric Training: https://www.skool.com/microsoft-fabri... #dataengineering #python #microsoftfabric In this video, I make use of Microsoft Fabric's data engineering experience - specifically the Synapse Data Engineering Notebooks (pySpark engine) to read a JSON file in our Lakehouse Files area, parse the JSON structure, clean the data a bit, transform some of the columns and then LOAD the data into a nice Lakehouse Table. This video follows on from my last video, in which I extracted data from an external weather API, using Data Pipelines in Microsoft Fabric, then loaded the raw JSON into Lakehouse Files.    • Extract and Load from External API to...   Next parts of the series include: Data validation of pySpark dataframes in Fabric using Great Expectations Visualising the weather data in Power BI. -LINK TO OPENWEATHERMAP- Here's the API used in this tutorial: https://openweathermap.org/api -OTHER VIDEOS YOU MIGHT LIKE- LAKEHOUSE -    • Microsoft Fabric Lakehouse Tutorial   Data warehouse -    • Microsoft Fabric Data Warehouse Expla...   Data pipelines -    • Three data pipeline use cases to make...   End-to-end data flows project -    • Dataflows end-to-end project (Microso...   OneLake -    • OneLake - the FIRST thing you need to...   -TIMELINE- 0:00 Intro 1:09 Recap on the Lakehouse Files 1:40 Intro to notebook structure 2:12 Reading JSON into pySpark dataframe 4:07 Exploring the data in Azure Storage Explorer & VS Code 5:28 Parsing the JSON 8:52 Datetime conversion 9:55 Calculated columns 11:25 Rounding numbers 12:37 Code refactoring 15:Load dataframe to Lakehouse table -LINKEDIN- Not following the LinkedIn page yet? Here's the link:   / learnmicrosoftfabric   -ABOUT WILL- Hi, I'm Will! I'm hugely passionate about data and using it to create a better world. I currently work as a Consultant, focusing on Data Strategy, Data Engineering and Business Intelligence (within the Microsoft/Azure/Fabric environment). I have previously worked as a Data Scientist. I started Learn Microsoft Fabric to share my learnings on how Microsoft Fabric works and help you build your career and build meaningful things in Fabric. -SUBSCRIBE- Not subscribed yet? You should! There are lots of new videos in the pipeline covering all aspects of Microsoft Fabric.