"Immutable Data Science with Datomic, Spark and Kafka" by Konrad Scorciapino and Mauro Lopes скачать в хорошем качестве

"Immutable Data Science with Datomic, Spark and Kafka" by Konrad Scorciapino and Mauro Lopes

We would like to present our company's approach to data science architecture, which is novel in its use of the Datomic database (in a microservices architecture) and the integration of Datomic with Spark. We leverage several unique properties of Datomic, Spark, and Kafka to achieve scalable real time analysis against production data without resorting to traditional ETL techniques: Immutability and consistent timeline: machine learning models in a microservice architecture with full traceability without copying data Multi-database querying: a fresh approach to OLAP that is both unified with and isolated from production OLTP services, and enables read auditing and granular access control for exploratory use cases without performance impact in production Modern stored procedures: a fresh take on exposing domain logic functions (e.g., canonical financial calculations in service code, under source control), to multi-database Datomic queries, enabling consistent definitions for data scientists and engineers Pass-by-reference queries: allow saving and sharing of data via immutable saved queries, which will never change for a given "as-of" timestamp, as well as persistent materialized views (Spark RDDs) stored in cluster memory or encrypted S3 Horizontal read scalability: parallel queries of live production data, and downstream transformation and analysis, sharded across an arbitrarily large Spark cluster (moving the code to the data) Immutable messaging substrate: Kafka supports real time Spark streaming use cases in addition to exploratory batch functionality This architecture is an alternative to the popular "lambda" and "kappa" architectures, and will be of primary interest to architects looking for innovation in modern technologies, engineers interested in Clojure, Datomic and Datalog, modelers hungry for data, and analysts making data-based decisions. In addition, we work with sensitive personal information that remains encrypted at rest, making our solution relevant for those interested in information security. In summary, this solution has allowed us to avoid ETL / database synchronization pipelines while preserving scalability and isolation of transactional and analytical use cases. Konrad Scorciapino NUBANK @konr Konrad Scorciapino is a data architecture engineer at Nubank. It was love at first sight with parentheses and his beautiful wife. He organizes the "Clojure São Paulo" and "Machine Learning São Paulo" user groups and his 2015 year resolution is to read 60 books (40 to go!) Mauro Lopes NUBANK @maurolopes23 Mauro is a data architecture engineer at Nubank. He holds a Master's degree in Graph Theory, plays badminton and has Erdős number 3.