Manage your data pipelines with Dagster | Software defined assets | IO Managers | Updated project скачать в хорошем качестве

Manage your data pipelines with Dagster | Software defined assets | IO Managers | Updated project

In this video we will revisit dagster. We will talk about changes to this workflow orchestration system due to recent updates (update from version 0.15 to 1.3.1) Dagster is an orchestrator that's designed for developing and maintaining data assets, such as tables, data sets, machine learning models, and reports. We will cover Software Defined Assets as dagster is pushing towards the Software Defined Assets. By default our pipeline outputs are stored as a pickle file in the dagster home folder. What if we want to store the outputs in a database table, or in a readable file such as a csv or parquet file. Dagster provides us with Input and Output managers (IO managers) that enable reading and writing data to storage systems. Using Store IO managers we can save the outputs in a file system or store our data as tables in a database. We will define file csv/parquet and database IO Managers. Link to previous video:    • Getting started with Dagster | Create...   Link to GitHub repo: https://github.com/hnawaz007/pythonda... Get started with Dagster in just three quick steps: Install Dagster, Define assets and Materialize assets. Create a virtual environment: python -m venv env Activate the virtual environment: env\Scripts\activate To install Dagster into an existing Python environment, run: pip install dagster dagit Command to create a new project dagster project scaffold --name my-dagster-project Additional libraries required: Pandas, psycopg2 To run dagster issue following command: dagit dagster-daemon run Access Dagit UI on port 3000: http://127.0.0.1:3000 💥Subscribe to our channel:    / haqnawaz   📌 Links ----------------------------------------- #️⃣ Follow me on social media! #️⃣ 🔗 GitHub: https://github.com/hnawaz007 📸 Instagram:   / bi_insights_inc   📝 LinkedIn:   / haq-nawaz   🔗   / hnawaz100   ----------------------------------------- #Python #ETL #Dagster Topics covered in this video: ================================== 0:00 - Introduction to Dagster 2:11 - Dagster create new project 3:03 - Dagster Project Structure 4:18 - Software Defined Assets 5:35 - Install Required Libraries 5:58 - Source DB Connection 6:27 - Source Asset 10:05 - File IO Manager 14:16 - Second Asset 16:19 - Parquet IO Manager 16:26 - Database IO Manager 19:05 - Materialize Assets