Sync Docs
Sync HomeLaunch GradientBook Demo
  • Sync Gradient
    • The Gradient Platform
      • How Does it Work?
    • Discover Quickstart
    • Add Workspace
      • Create Sync API Key
      • Add Databricks Workspace
        • AWS Databricks Setup
          • EventBridge Setup
        • Azure Databricks Setup
      • Webhook Setup
    • Project Setup
      • Import Jobs to Projects
      • Verify and Run Jobs
      • Generate and Apply Recommendation
    • Advanced Use Cases
      • Install the Sync-CLI
      • Manual Workspace Setup
        • AWS Instance Profile
      • Apache Airflow for Databricks
      • Gradient Terraform Integration
    • Project Settings
    • Account Settings
    • ROI Reporting
    • FAQ
  • Tutorials & Best Practices
    • Running Gradient in Production
      • Production Auto-Enabled
      • Optimization Windows
      • Development Clones
    • Demos
  • Developer Docs
    • Resources
    • Sync Python Library
    • Gradient CLI Walkthrough
  • Security
    • Privacy and Security Compliance
  • Trust Center
    • Portal
  • Product Announcements
    • Product Updates
  • Need Help?
    • Troubleshooting Guide
Powered by GitBook
On this page
  • Introduction
  • Demo Video
  • What problem is Gradient solving?
  • Who is it for?

Was this helpful?

Export as PDF
  1. Sync Gradient

The Gradient Platform

NextHow Does it Work?

Last updated 10 months ago

Was this helpful?

Introduction

Gradient is an infrastructure management and optimization platform that continuously monitors and learns users' Databricks Jobs so that it can optimize the data infrastructure to hit cost and runtime goals. It supports both co-pilot and autopilot modes. Use it as a co-pilot to receive passive recommendations for optimizations you can apply in a click, or enable auto-apply for optimization at scale.

Gradient uses a closed-loop feedback system to automatically build custom tuned machine learning models for each Databricks Job it is managing, using historical run logs. Through this mechanism, Gradient continuously drives Databricks Jobs cluster configurations to hit user defined business goals, such as maximum costs and runtimes.

Demo Video

What problem is Gradient solving?

Managing and optimizing Databricks Job clusters is tedious, time intensive, and difficult for data and platform engineers; there are far too many Spark configurations and infrastructure choices to know what's right and what makes sense. Additionally, just when they've gone through the effort of optimizing a job, something changes that wipes away all that hard work.

To make matters worse, changing infrastructure incorrectly can also lead to crashed jobs due to out of memory errors. A major risk to production pipelines that often blocks engineers from optimizing in the first place.

If engineers do want to try managing clusters, it comes at the expense of taking time away from delivering new products and features. Furthermore, managing at scale where hundreds or thousands of jobs are running is simply not feasible for any sized team.

Gradient provides data teams with an easy and scalable solution that can significantly diminish engineering time spent on cluster optimization, while cutting costs and improving runtimes. It can even automatically manage clusters for all of your jobs - with no code changes.

Who is it for?

  • Data Engineers - Avoid spending time tuning and optimizing clusters while still achieving optimal cost and runtime performance.

  • Data Platform Managers - Ensure your team's Databricks Jobs are achieving high level business objectives without having to bug your engineers or change any code. This becomes particularly important for teams who are looking to scale their Databricks usage.

  • VP of Engineering / CTOs - Gradient works for you and not the cloud providers. It was built to help you efficiently produce data products that meet your business.