Sync Docs
Sync HomeLaunch GradientBook Demo
  • Sync Gradient
    • The Gradient Platform
      • How Does it Work?
    • Discover Quickstart
    • Add Workspace
      • Create Sync API Key
      • Add Databricks Workspace
        • AWS Databricks Setup
          • EventBridge Setup
        • Azure Databricks Setup
      • Webhook Setup
    • Project Setup
      • Import Jobs to Projects
      • Verify and Run Jobs
      • Generate and Apply Recommendation
    • Advanced Use Cases
      • Install the Sync-CLI
      • Manual Workspace Setup
        • AWS Instance Profile
      • Apache Airflow for Databricks
      • Gradient Terraform Integration
    • Project Settings
    • Account Settings
    • ROI Reporting
    • FAQ
  • Tutorials & Best Practices
    • Running Gradient in Production
      • Production Auto-Enabled
      • Optimization Windows
      • Development Clones
    • Demos
  • Developer Docs
    • Resources
    • Sync Python Library
    • Gradient CLI Walkthrough
  • Security
    • Privacy and Security Compliance
  • Trust Center
    • Portal
  • Product Announcements
    • Product Updates
  • Need Help?
    • Troubleshooting Guide
Powered by GitBook
On this page
  • Before you start
  • Step 1: Go to Org Settings > Integrations page > Add > Databricks Workspace
  • Step 2: Databricks Workspace details and Sync API key
  • Step 3: Cloud provider, logs, and data collection

Was this helpful?

Export as PDF
  1. Sync Gradient
  2. Add Workspace

Add Databricks Workspace

PreviousCreate Sync API KeyNextAWS Databricks Setup

Last updated 8 months ago

Was this helpful?

The Databricks workspace setup is a one-time setup for your organization. With the webhook tutorial below, all users within an organization will be able to:

  • Onboard new jobs onto Gradient with a single click through the Gradient UI

  • Onboard jobs at mass scale

  • Integrate Gradient without any modifications to your Databricks workflows tasks

Before you start

Setting up a Databricks Workspace integration involves three steps:

  1. Giving Gradient access to your Databricks Workspace (such as Databricks host, token, and other details)

  2. Giving Gradient access to your cloud provider to fetch metadata on compute infra (such as EC2 instances, EBS volumes, etc)

  3. Configuring a Databricks Webhook to notify Gradient about your job start and stop events

This doc covers the first step. At the bottom of this doc is a link to the doc that will help you complete the second step.

Step 1: Go to Org Settings > Integrations page > Add > Databricks Workspace

In the integrations page, click on "ADD" to see the "Add Databricks Workspace" console.

Step 2: Databricks Workspace details and Sync API key

We need to know how to connect to your Databricks Workspace. Provide details of your Databricks Workspace and choose the Sync API Key to use with this workspace integration.

  • Databricks Workspace ID - Can be found in the address bar of your web browser in your Databricks URL. It is the "o" parameter "o=9172567527460388", so in this case you would enter the number "9172567527460388".

  • Databricks Host - Can be found in the address bar of your web browser when at your Databricks workspace. It should look like this: https://dbc-6c213588-2400.cloud.databricks.com/

  • Sync API Key - Select the Sync API key from the drop down menu. If you haven't created one yet, you can create one here.

  • Databricks Plan Type - Select your plan type which will impact the pricing used to calculate your Databricks costs.

Step 3: Cloud provider, logs, and data collection

We need to know how to get logs and collect data for your job runs.

Select one of the options below to continue setting up your Databricks Workspace integration depending on your cloud provider.

Databricks Token - You will need Databricks admin access to generate a for your Databricks workspace. Copy and paste your token value here. We recommend setting the "Lifetime" field blank so the token does not expire and interrupt any service down the line.

personal access token
Cover

AWS Databricks Setup

Cover

Azure Databricks Setup