Real Interview Q&A for Senior Data Engineer #3 | Migration project | Surfalytics скачать в хорошем качестве

Real Interview Q&A for Senior Data Engineer #3 | Migration project | Surfalytics

In this interview, I discuss my background as a data engineer, working with companies like Amazon and Microsoft on various data-intensive projects. I dive deep into my experience with cloud data platforms like Databricks and Snowflake, highlighting a successful Delta Lake implementation on Databricks and addressing the challenges of managing costs in Snowflake. I also share my insights on Redshift, its evolution, and the critical factors to consider when evaluating tools for data migration projects, emphasizing the importance of understanding business logic and decoupling solutions. Specifically, I cover the following key points: Core skills in Snowflake, DBT, and Databricks, with experience across major cloud providers. The successful implementation of Delta Lake on Databricks at Microsoft, overcoming the challenges of a legacy on-premises data warehouse. Recent projects, including a fintech project involving Trino on Kubernetes and a Snowflake migration for Rogers. Evaluating tools and technologies, particularly in the context of large organizations and migration projects. My experience with Redshift and BigQuery, highlighting a complex Oracle-to-Redshift migration at Amazon. Key considerations for optimizing Redshift tables and managing concurrency issues. A comparison of Redshift and Snowflake, focusing on cost management and performance optimization. My approach to data migration, including understanding business logic, decoupling solutions, and conducting POCs. If you're looking to learn more about cloud data warehousing, migration strategies, and the nuances of platforms like Databricks, Snowflake, and Redshift, this interview offers valuable insights from a seasoned data engineer. Thank you for watching our video "Real Interview Q&A for Senior Data Engineer | Migration project | Surfalytics" on SurfalyticsTV as part of a series of "Real Interviews records", click the link to watching the previous videos:    • Real Interview Q&A for Senior Data Enginee...   Subscribe for channel for more videos:    / @surfalyticstv   Interested in learning more about data engineering best practices and overcoming real-world challenges? Join our community at https://surfalytics.com/ to connect with other data professionals and stay up-to-date on the latest trends in the field. #dataengineering #clouddatawarehousing #datamigration #databrick #snowflake #redshift #trino #kubernetes #etl #bigquery #aws #azure #gcp #devops #opensource #costoptimization #performanceoptimization #medallionarchitecture #deltalake Timecode: 00:00 - Intro 00:23 - Tell about yourself 01:35 - Rock Your Data experience 03:45 - The biggest challenge 06:45 - Team dynamics 07:19 - How to evaluate tools and technologies 11:03 - BigQuery and Redshift 14:16 - Design constipation 19:54 - Why Snowflake is better than Redshift? 20:54 - Cluster utilisation 21:46 - Python practice 27:46 - SQL practice 36:52 - My questions ================= What is Surfalytics? Inspired by West Coast surfing spots 🏖️ and Pacific Ocean vibes 🌊. Created to help you start a new career in the data analytics space, and develop data engineering and analytics skills through coaching. It will teach you not just dry skills, but will keep your focus on delivering significant value to businesses in the analytics realm as well as help to get fair compensation 💰 for the work you’re passionate about ❤️‍🔥. The goal of Surfalytics is to assist you in achieving one of the following: 🏄‍♂️ Land your first job in the data industry with literally zero experience. I have accomplished this many times across the globe. 🏄 Advance from a middle-level role to a senior position (as an Analyst or Engineer). 🏄‍♀️ Transition from a non-technical Analyst role to a technical Engineer role. Moreover, we will focus on creating a highly competitive CV and securing top job offers. We will not consider any lowball offers, focusing only on top-tier companies and well-paid opportunities. Finally, Surfalytics is a results-driven community with a very narrow focus, resulting in a high return on investment (ROI). Here, ‘investment’ does not mean money but your time. I am literally fighting for your attention to encourage you to study and work hard, instead of watching Netflix or playing video games. This is the best YouTube channel for Data Analytics and Engineering. You will patch up a lack of knowledge and get new experience and tips to build a Data Analyst roadmap or Data Engineer roadmap for yourself. Want to be part of our growing community? Join on Surfalytics.com #surfalytics #dmitryanoshin #datacommunity #freecourses #dataanalysis #dataengineering #roadmap #careerpath #mindmaps #tools #overview #dataanalysttips