Building Production-Grade AI Systems at MasterClass скачать в хорошем качестве

Building Production-Grade AI Systems at MasterClass

Join Aman Gupta (MasterClass) and Hugo Bowne-Anderson for a fireside chat on building and evaluating production-grade LLM systems. Drawing from the MasterClass team’s experience developing interactive, voice-based AI products with branded instructor personas, we’ll explore how post-training, evaluation, and infrastructure choices shape the next generation of applied AI. Key Topics of Discussion Building Production-Grade LLM Systems at MasterClass: How ML and AI became one of the highest-leverage functions for enhancing the experience for MasterClass—and why they chose to build in-house ML systems instead of just using off-the-shelf APIs. From Prototype to Production Reality: Moving beyond basic LLM integrations to build robust, scalable ML systems that serve millions of learners. Post-Training Excellence: What "post-training" really means in practice and why it's become a critical differentiator to go from AI wrapper to an AI-native product. Infrastructure Wisdom: How tools like Metaflow can power sophisticated post training pipelines that adapt as AI technology evolves. The Hard Problems: Evaluating AI systems for real product impact—connecting technical metrics to business outcomes like user engagement and retention. Safety & Alignment: Practical approaches to building AI systems that align with product and user goals, including data patterns that help with alignment training. LLMs writing prompts: How to use AI to improve AI - using automated prompt engineering.. The Future of ML Systems: What's next in post-training, evaluation methodologies, and building differentiated AI products. This session is ideal for ML engineers, AI product teams, and technical leaders facing the challenge of moving AI systems from experiments to production. Whether you're dealing with model evaluation challenges, infrastructure decisions, or aligning AI with business goals, you'll gain practical insights from real-world implementation. 00:00 Introduction and Greetings 01:55 Aman's Role at Masterclass 02:35 Masterclass Overview and AI Integration 05:00 Machine Learning and AI at Masterclass 07:50 In-House vs. Third-Party Models 13:35 Post-Training Techniques 23:34 Challenges in AI Model Deployment 31:32 Evaluating Models for User Value 32:58 Masterclass Case Study: User Research and Prototyping 35:23 Mixed Methods Research: Combining Qualitative and Quantitative 37:13 Practical Evaluation Techniques 39:27 Choosing Tools and Designing a Flexible Stack 45:34 The Role of Metaflow in Machine Learning Workflows 51:02 Aligning AI with Product and User Goals 56:09 Advice for Building Robust AI Products 01:00:20 Conclusion and Final Thoughts