What is Optimized?

Spark Cluster Configurations

The current continuous optimization algorithm modifies the Apache Spark configurations below.
Cluster Configurations
  • Driver node instance type
  • Worker node instance type
  • Number of workers (fixed cluster)
  • Autoscaling min and max workers
Spark Configurations
  • spark.executor.instances
  • spark.executor.memory
  • spark.driver.memory
  • spark.memory.offHeap.enabled
  • spark.executor.memory
  • spark.executor.cores
Note: Future upgrades of the core algorithm may expand the scope of the optimization parameters

Real-time Spot Instance Information

If the Spot market is used, real-time data of Spot pricing and availability is pulled at the time of the recommendation generation.