Visual Flow, open source ETL tool powered by Apache Spark Dmitri Pavlov, Alexander Shevtsov скачать в хорошем качестве

Visual Flow, open source ETL tool powered by Apache Spark Dmitri Pavlov, Alexander Shevtsov

A presentation from ApacheCon@Home 2021 https://apachecon.com/acah2021 We want to present Visual Flow – a cloud-native open-source ETL tool based on Apache Spark unified analytics engine with a Graphical User Interface. It combines the best features of Spark, Kubernetes and Argo into a single system with the following features: Portability, flexibility and multi-cloud compatibility Increased developer productivity High availability, performance and fault tolerance Cost-effective Open-source Visual Flow provides a Graphical User Interface to create ETL jobs and combine them in Data Processing Pipelines, run/schedule them, and monitor execution. It is a multi-tenant system. You can create multiple projects in a single cluster. Each project is represented as a namespace, which allows allocating resources and providing access to the users in different roles. Visual Flow does not require a database. It is possible to create all objects as native Kubernetes resources. Visual Flow leverages Kubernetes authorization to manage users and their roles within a project. Visual Flow provides the ability to create parameters (e.g., connection info) and reuse them in Jobs.