Spark Performance Tuning | Practical Performance Optimization | Spark Developer скачать в хорошем качестве

Spark Performance Tuning | Practical Performance Optimization | Spark Developer

Dynamic Allocation Property set to true by default is false also dynamic initial and max executors should be set: spark.dynamicAllocation.enabled= true spark.dynamicAllocation.minExecutors=10, spark.dynamicAllocation.maxExecutors= 30, Executor and Driver memory need to be set: spark.executor.memory=2g, spark.driver.memory=2g, Example of spark-submit spark-submit --master yarn-cluster \ --driver-cores 2 \ --driver-memory 2G \ --num-executors 10 \ --executor-cores 5 \ --executor-memory 2G \ --conf spark.dynamicAllocation.minExecutors=5 \ --conf spark.dynamicAllocation.maxExecutors=30 \ --conf spark.dynamicAllocation.initialExecutors=10 \ # same as --num-executors 10 --class com.spark.sql.jdbc.SparkDFtoOracle2 \ Spark-hive-sql-Dataframe-0.0.1-SNAPSHOT-jar-with-dependencies.jar