The History and the Future of Run-time Compilation in MXNet скачать в хорошем качестве

The History and the Future of Run-time Compilation in MXNet

Dr. Przemyslaw Tredak, Sr. DL Frameworks Engineer @ NVIDIA As the computational capabilities of Deep Learning hardware become greater, there is a growing performance discrepancy between different types of DL operations. Some operations, like convolutions and fully connected layers, are in the spotlight for being compute intensive and are therefore benefiting greatly from those hardware advancements. Other operations, like normalization layers or even simple ReLU activations, fly under most people's radar when thinking about model performance optimization. However, because advancements in hardware focus on central well-known model operations, those traditionally small parts of the model are now taking a significantly greater portion of the training time. For example, when we started optimizing the ResNet 50 model for the first round of MLPerf benchmark, ReLU activation took about 10% of the training time. In this talk we will present how RTC (RunTime Compilation) lets us tackle this problem within MXNet currently. We will also talk about the work being done to expand RTC use in the upcoming MXNet 2.0 to achieve the best efficiency.