Too Big to Train: Large model training in PyTorch with Fully Sharded Data Parallel скачать в хорошем качестве

Too Big to Train: Large model training in PyTorch with Fully Sharded Data Parallel

With the popularity of Large Language Models and the general trend of scaling up model and dataset sizes comes challenges in training. Despite hardware improvements, many models are too large to fit onto a single GPU or large enough that small batch sizes lead to long training times. One strategy for parallelizing training is Fully Sharded Data Parallel (FSDP), provided by PyTorch. This strategy splits models into shards and distributes shards across parallel GPUs. This strategy can be used to train very large models and to scale up training. In this talk, we'll discuss implementing FSDP in your training code, examine training performance from an efficiency perspective and compare with another parallelization strategy, data parallelism. Some experience with Python, PyTorch and deep learning is expected. _______________________________________­________ This webinar was presented by Collin Wilson (SHARCNET) on April 9th, 2025, as a part of a series of weekly Compute Ontario Colloquia. The webinar was hosted by SHARCNET. The colloquia cover different advanced research computing (ARC) and high performance computing (HPC) topics, are approximately 45 minutes in length, and are delivered by experts in the relevant fields. Further details can be found on this web page: https://www.computeontario.ca/trainin... . Recordings, slides, and other materials can be found here: https://helpwiki.sharcnet.ca/wiki/Onl... SHARCNET is a consortium of 19 Canadian academic institutions who share a network of high performance computers (http://www.sharcnet.ca). SHARCNET is a part of Compute Ontario (http://computeontario.ca/) and Digital Research Alliance of Canada (https://alliancecan.ca).