Optimization Windows

Step 1: Decide on an optimization window

Select an optimization window timeframe (e.g. first week of the month). By limiting the optimization window to a finite, but periodic, time frame, it allows engineers to still be in control with what the optimization is doing.

Step 2: Select jobs to optimize

Select N number of jobs you want to optimize that are good candidates for tuning in your PROD environment.

By good candidates we mean you're OK if there's some variance in runtime and cost in your PROD environment during the optimization phase.

During the optimization Gradient will try different configurations and runtime/cost may go up or down during this period. Be sure that the stakeholders of your jobs are OK with this.

Step 3: Enable auto-apply

Enable auto-apply for those jobs to allow Gradient to automatically update your jobs during the optimization window. Let your engineers check-in on those jobs during the window via the Gradient UI to monitor progress.

Step 4: Pick a configuration and lock it in

At the end of the time frame, pick the configuration that lead to a cost and runtime you prefer. Apply those settings in your Databricks jobs, and disable auto-apply in the Gradient UI to lock in the configurations and prevent future changes.

Step 5: Go back to step 2

When a new optimization window arrives, go back to step 2 and select a new batch of jobs to optimize.