The Gradient Platform

Introduction

Gradient is an infrastructure management and optimization platform that continuously monitors and learns users' Databricks Jobs to automatically manage infrastructure to hit cost and runtime goals.

Using a closed-loop feedback system, Gradient automatically builds custom tuned machine learning models for each Databricks Job it is managing using historical run logs. Through this mechanism, Gradient continuously drives Databricks Jobs cluster configurations to hit user defined business goals.

Demo Video

What problem is Gradient solving?

Managing and optimizing Databricks Job clusters is tedious, time intensive, and difficult for data and platform engineers; there are far too many Spark configurations and infrastructure choices to know what's right and what makes sense. Additionally, just when they've gone through the effort of optimizing a job, something changes that wipes away all that hard work.

To make matters worse, changing infrastructure incorrectly can also lead to crashed jobs due to out of memory errors. A major risk to production pipelines that often blocks engineers from optimizing in the first place.

If engineers do want to try managing clusters, it comes at the expense of taking time away from delivering new products and features. Furthermore, managing at scale where hundreds or thousands of jobs are running is simply not feasible for any sized team.

Gradient provides data teams with an easy and scalable solution that automatically manages clusters for all jobs - with no code changes.

Who is it for?

• Data Engineers - Avoid spending time tuning and optimizing clusters while still achieving optimal cost and runtime performance.

• Data Platform Managers - Ensure your team's Databricks Jobs are achieving high level business objectives without having to bug your engineers or change any code. This becomes particularly important for teams who are looking to scale their Databricks usage.

• VP of Engineering / CTOs - Gradient works for you and not the cloud providers, to efficiently produce data products that meet your business needs.