How Does it Work?
Gradient monitors all your Spark workloads in order to recommend the best configuration based on your requirements. To achieve this, Gradient uses Sync's proprietary Autotuner engine, which analyzes the event logs for your workloads and predicts how they will perform with various configurations. The best configuration is then sent to a Sync Project for you to review.
Full Gradient integration utilizes the Sync Python library to deliver recommendations to your Sync Projects.
Sync Projects is the fully featured continuous optimization solution and is a way to organize and access all the recommendations for each of your configured workloads. Projects also unlock more optimization potential and key features which may be important for your infrastructure. Each Project is continually updated with the most recent recommendation provided by Gradient Autotuner engine, allowing you to review run level efficiency metrics as well as aggregated metrics over time for the configured Spark workload.
Gradient is available at no charge though the Community Edition of the platform. Organizations that require (TBD) should choose the Enterprise edition of Gradient.
- Auto Databricks jobs import - Provide your Databricks token, and we’ll do all the heavy lifting of automatically fetching all of your qualified jobs and import them into Gradient.
- You set max runtime, Sync minimizes costs - Simply set your ideal max runtime and we’ll configure the cluster to hit your goals at the lowest cost.
- Continuous monitoring and optimization - Gradient continuously monitors your clusters metrics after each job run to both let you know of new inefficiencies and to help aggregate data to inform a better recommendation for next time.
- Workflow integration with Airflow & Databricks Workflows - New python libraries and quickstart tutorials for Airflow and Databricks Workflows make it easy to integrate Gradient into your favorite orchestrators.
- Databricks autoscaling optimization - Optimize your min and max workers for your job. It turns out autoscaling parameters are just another set of numbers that need tuning. Check out our previous blog post.
Last modified 2d ago