Resource-Efficient Deep Learning Execution - Deepak Narayanan | Stanford MLSys #50 скачать в хорошем качестве

Resource-Efficient Deep Learning Execution - Deepak Narayanan | Stanford MLSys #50

Episode 50 of the Stanford MLSys Seminar Series! Resource-Efficient Execution of Deep Learning Computations Speaker: Deepak Narayanan Abstract: Deep Learning models have enabled state-of-the-art results across a broad range of applications; however, training these models is extremely time- and resource-intensive, taking weeks on clusters with thousands of expensive accelerators in the extreme case. In this talk, I will describe two ideas that help improve the resource efficiency of model training. In the first half of the talk, I will discuss how pipelining can be used to accelerate distributed training. Pipeline parallelism facilitates model training with lower communication overhead than previous methods while still ensuring high compute resource utilization. Pipeline parallelism also enables the efficient training of large models that do not fit on a single worker; for example, we used pipeline parallelism at Nvidia to efficiently scale training to language models with a trillion parameters on 3000+ GPUs. In the second half of this talk, I will describe how resources in a shared cluster with heterogeneous compute resources (e.g., different types of hardware accelerators) should be partitioned among different users to optimize objectives specified over one or more training jobs. Heterogeneity-aware scheduling can improve various scheduling objectives, such as average completion time, makespan, or cloud computing resource cost, by up to 3.5x. Bio: Deepak is a Senior Researcher in the Systems group at Microsoft Research Redmond. His broad research interests are in distributed systems and systems for Machine Learning. He graduated from Stanford with a Ph.D. in Computer Science in September 2021, where he was advised by Prof. Matei Zaharia. -- 0:00 Presentation 30:21 Discussion Stanford MLSys Seminar hosts: Dan Fu, Karan Goel, Fiodar Kazhamiaka, and Piero Molino Executive Producers: Matei Zaharia, Chris Ré Twitter:   / realdanfu​     / krandiash​     / w4nderlus7   -- Check out our website for the schedule: http://mlsys.stanford.edu Join our mailing list to get weekly updates: https://groups.google.com/forum/#!for... #machinelearning #ai #artificialintelligence #systems #mlsys #computerscience #stanford #megatron #microsoftresearch #microsoft