Cluster Serving: Distributed Model Inference using Apache Flink in Analytics Zoo скачать в хорошем качестве

Cluster Serving: Distributed Model Inference using Apache Flink in Analytics Zoo

As deep learning projects evolve from experimentation to production, there is increasing demand to deploy deep learning models for large-scale, real-time distributed inference. While there are many tools available for relevant tasks (such as model optimization, serving, cluster scheduling, workflow management, etc.), it is still a challenging process for many deep learning engineers and scientists to develop and deploy distributed inference workflow that can scale out to large clusters in a transparent fashion. To address this challenge, we have developed Cluster Serving on Apache Flink, an automated and distributed serving solution that supports a wide range of deep learning models (such as TensorFlow, PyTorch, Caffe, BigDL, and OpenVINO). It provides simple publish-subscribe (pub/sub) and REST APIs, through which users can easily send their inference requests to the input queue using simple Python or HTTP APIs. Cluster Serving will then automatically manage the scale-out and real-time model inference across a large cluster, using Apache Flink. In this video, we present the architecture design for Cluster Serving, and discuss the underlying design patterns and tradeoffs to deploy deep learning models on distributed Big Data streaming frameworks in production. In addition, we will also share real-world experience and "war stories" of users who have adopted Cluster Serving on Apache Flink to develop and deploy distributed inference workflow. Outline • Motivation: It will discuss the real-world challenges in developing and deploying distributed inference workflow. • Architecture: It will present the architecture design for Cluster Serving, and the underlying design patterns and tradeoffs to deploy deep learning models on Apache Flink in production • Programming Model: It will describe the pub/sub API of Cluster Serving (which provides a simple abstraction for the underlying distributed streaming inference), and how it can be used to implement synchronous RESTFUL APIs for the users. • Real-World Experience: It will present real-world use cases, experience and "war stories" of users who have adopted Cluster Serving to develop and deploy distributed inference workflow.