Welcome to Gradient
Gradient is an optimization platform that continuously learns and makes recommendations for your Databricks Jobs clusters with the objective of achieving your desired cost or runtime goals.
By utilizing a "closed-loop" feedback system based on historical data of each job, Gradient automatically builds custom tuned models for each workload it controls. As a result, Gradient continuously drives the Databricks Jobs cluster configurations to hit user defined goals.
The diagram below shows the high level flow of information starting from running a Databricks Job in a client environment to updating a Spark cluster configuration.
System diagram of how Gradient works
Optimizing Databricks Jobs clusters for cost and time is a time consuming and difficult task for any data engineer. Furthermore, cloud infrastructure continuously changes, with updated code, different data, and spot pricing / availability. For users with many jobs, keeping up with optimization at scale is simply not possible.
We created Gradient to provide data teams with a an easy, scalable solution to ensure their workloads remain optimized, without having specialized knowledge of Spark internals and without having to modify any logic within the workload code.
- Sr. Data Engineers - Avoid spending time tuning and optimizing clusters while still achieving optimal cost and runtime performance.
- Data Platform Managers - Ensure your team's Databricks Jobs are achieving high level business objectives without having to bug your engineers or change any code. This becomes particularly important for teams who are looking to scale their Databricks usage.
- VP of Engineering / CTOs - Gradient works for you and not the cloud providers, to continuously monitor and optimize your Databricks workloads to ensure optimal ROI.
Last modified 2d ago