Alluxio Data Orchestration for Machine Learning скачать в хорошем качестве

Alluxio Data Orchestration for Machine Learning

Alluxio Day III April 27, 2021 For more on Alluxio Day: https://www.alluxio.io/alluxio-day/ For more Alluxio events: https://alluxio.io/events/ Speakers: Lu Qiu (Software Engineer, Alluxio) Bin Fan (Founding Engineer & VP of Open Source, Alluxio) Alluxio’s capabilities as a Data Orchestration framework have encouraged users to onboard more of their data-driven applications to an Alluxio powered data access layer. Driven by strong interests from our open-source community, the core team of Alluxio started to re-design an efficient and transparent way for users to leverage data orchestration through the POSIX interface. This effort has a lot of progress with the collaboration with engineers from Microsoft, Alibaba and Tencent. Particularly, we have introduced a new JNI-based FUSE implementation to support POSIX data access, created a more efficient way to integrate Alluxio with FUSE service, as well as many improvements in relevant data operations like more efficient distributedLoad, optimizations on listing or calculating directories with a massive amount of files, which are common in model training. We will also share our engineering lessons and roadmap in future releases to support Machine Learning applications.