What is Optimized?
The current continuous optimization algorithm modifies the Apache Spark configurations below.
Cluster Configurations
- Driver node instance type
- Worker node instance type
- Number of workers (fixed cluster)
- Autoscaling min and max workers
Spark Configurations
- spark.executor.instances
- spark.executor.memory
- spark.driver.memory
- spark.memory.offHeap.enabled
- spark.executor.memory
- spark.executor.cores
Note: Future upgrades of the core algorithm may expand the scope of the optimization parameters
If the Spot market is used, real-time data of Spot pricing and availability is pulled at the time of the recommendation generation.
.
Last modified 2d ago