Add Databricks Workspace
The Databricks workspace setup is a one-time setup for your organization. With the webhook tutorial below, all users within an organization will be able to:
Onboard new jobs onto Gradient with a single click through the Gradient UI
Onboard jobs at mass scale
Integrate Gradient without any modifications to your Databricks workflows tasks
Before you start
Setting up a Databricks Workspace integration involves three steps:
Giving Gradient access to your Databricks Workspace (such as Databricks host, token, and other details)
Giving Gradient access to your cloud provider to fetch metadata on compute infra (such as EC2 instances, EBS volumes, etc)
Configuring a Databricks Webhook to notify Gradient about your job start and stop events
This doc covers the first step. At the bottom of this doc is a link to the doc that will help you complete the second step.
Step 1: Go to Org Settings > Integrations page > Add > Databricks Workspace
In the integrations page, click on "ADD" to see the "Add Databricks Workspace" console.
Step 2: Databricks Workspace details and Sync API key
We need to know how to connect to your Databricks Workspace. Provide details of your Databricks Workspace and choose the Sync API Key to use with this workspace integration.
Databricks Workspace ID - Can be found in the address bar of your web browser in your Databricks URL. It is the "o" parameter
"o=9172567527460388", so in this case you would enter the number"9172567527460388".Databricks Host - Can be found in the address bar of your web browser when at your Databricks workspace. It should look like this:
https://dbc-6c213588-2400.cloud.databricks.com/Databricks Token - You will need Databricks admin access to generate a personal access token for your Databricks workspace. Copy and paste your token value here. We recommend setting the "Lifetime" field blank so the token does not expire and interrupt any service down the line.
Sync API Key - Select the Sync API key from the drop down menu. If you haven't created one yet, you can create one here.
Databricks Plan Type - Select your plan type which will impact the pricing used to calculate your Databricks costs.
Step 3: Cloud provider, logs, and data collection
We need to know how to get logs and collect data for your job runs.
Select one of the options below to continue setting up your Databricks Workspace integration depending on your cloud provider.
Last updated
