IEEE CIS Webinar: Scaling Challenges in LLM Training скачать в хорошем качестве

IEEE CIS Webinar: Scaling Challenges in LLM Training

The session will cover the challenges associated with scaling the training of large language models (LLMs). As LLMs grow in both size and complexity, the demands on computational resources, infrastructure, and cost management have escalated significantly. We will explore the various hurdles that researchers and engineers face when scaling LLMs, including the need for vast computational power, memory requirements, and the complexities of parallel processing architectures. The talk will delve into innovative solutions such as model parallelism, which allows for distributing model parameters across multiple machines, and mixed precision training, which reduces the memory and computation requirements by utilizing different numerical precisions. We will also discuss the role of efficient data loading and batching techniques that can significantly enhance training speed and model throughput. By examining these technical strategies alongside real-world case studies, attendees will gain a comprehensive understanding of how to effectively scale up the training processes for LLMs to achieve better performance and efficiency. Speaker Biography: Lalit Chourey is a lead software engineer with 11+ years of experience, focused on AI infrastructure at Meta. He specializes in designing and developing scalable systems for training large language models, leveraging extensive GPU arrays to significantly shorten training durations. At Meta, his work in real-time monitoring of AI training systems has played a critical role in Meta’s LLM training. Previously, Lalit was a senior software engineer at Microsoft, working on building large scale cloud service on Azure cloud platform.