Flux: Next-Generation Workload Management for High Performance Computing and Cloud скачать в хорошем качестве

Flux: Next-Generation Workload Management for High Performance Computing and Cloud

Daniel Milroy Computer Scientist Vanessa Sochat Computer Scientist LLNL (Lawrence Livermore National Laboratory) Abstract: Flux is a next-generation workload manager, the primary means to run workloads on the first NNSA1 exascale system El Capitan that was just announced as the #1 machine on the Top500 list at the Supercomputing Conference 2024, held in Atlanta Georgia. In total, Flux is deployed as the system scheduler on the following systems in the Top500: El Capitan (#1) Tuolumne (# 10) El Dorado (#20) rzAdams (#49) Tioga (#222) Tanaya (#299) Flux is unique in that it combines graph-based scheduling with fully hierarchical resource management, and enables portability, performance, and manageability of complex workflows. Along with serving the user-base of the national labs, Flux is making its mark on centers internationally2, and is a predominant player in the emerging space of HPC technologies in the cloud, offering Kubernetes operators and custom schedulers. With this emergence, the time is now to provide instruction to the larger community to learn about how Flux addresses the scheduling and fine-grained resource management of present day, complex scientific workflows that require interplay between simulation, analysis and visualization, and services. Given the flexibility to run as a central scheduler, under another workload manager, on a local machine, or in a container, Flux is a vehicle that promises portability of workflows not just between traditional centers, but also between centers and cloud. This workshop will provide practical demonstrations for interacting with Flux, starting with background on the project and proceeding through present day, innovative work that spans the space of HPC and cloud. More information: https://hpckp.org/