Sync Docs
Sync HomeLaunch GradientBook Demo
  • Sync Gradient
    • The Gradient Platform
      • How Does it Work?
    • Discover Quickstart
    • Add Workspace
      • Create Sync API Key
      • Add Databricks Workspace
        • AWS Databricks Setup
          • EventBridge Setup
        • Azure Databricks Setup
      • Webhook Setup
    • Project Setup
      • Import Jobs to Projects
      • Verify and Run Jobs
      • Generate and Apply Recommendation
    • Advanced Use Cases
      • Install the Sync-CLI
      • Manual Workspace Setup
        • AWS Instance Profile
      • Apache Airflow for Databricks
      • Gradient Terraform Integration
    • Project Settings
    • Account Settings
    • ROI Reporting
    • FAQ
  • Tutorials & Best Practices
    • Running Gradient in Production
      • Production Auto-Enabled
      • Optimization Windows
      • Development Clones
    • Demos
  • Developer Docs
    • Resources
    • Sync Python Library
    • Gradient CLI Walkthrough
  • Security
    • Privacy and Security Compliance
  • Trust Center
    • Portal
  • Product Announcements
    • Product Updates
  • Need Help?
    • Troubleshooting Guide
Powered by GitBook
On this page
  • Step 1: Select instance profile permissions and logging
  • Step 2: Verify the job modifications for Gradient
  • Step 3: Run your job!

Was this helpful?

Export as PDF
  1. Sync Gradient
  2. Project Setup

Verify and Run Jobs

PreviousImport Jobs to ProjectsNextGenerate and Apply Recommendation

Last updated 11 months ago

Was this helpful?

To ensure a successful operation of Gradient, there are a few final steps and checks that will depend on how you currently configure your jobs.

After these steps, you should have a successful first run of Gradient. If you need any help, feel free to reach out to us via intercom or email us at .

For the Databricks Job you want to optimize, go to the Databricks console Job's page.

Step 1: Select instance profile permissions and logging

Instance profiles: Users must have an instance profile correctly configured with access to AWS's S3 and describe cluster functions. See the instructions for more information.

Enable Logging: Logging must be enabled for your existing job, either DBFS or S3 log location is fine

Step 2: Verify the job modifications for Gradient

The following items should have been automatically completed via the job import step. These are just steps to verify the setup completed correctly. Ideally, no further action is required.

Webhook notification: Ensure that the job has webhooks enabled.

Spark environmental variables: Ensure the Spark environmental variables are populated with secrets.

Cluster init scripts: Ensure the Sync init script is selected

Step 3: Run your job!

Your Databricks job should be fully connected to Gradient. Simply run your job as you normally would via the Databricks UI and click on "Run"

After your job is completed, a secondary "Record Job Run" job will be started which will automatically collect the logs generated and transmit them to Gradient.

The "Record Job Run" job can take 10-15 minutes to complete.

After the completion of the "Record Job Run" job, head over to the Gradient UI, where you should see the first datapoint populated:

support@synccomputing.com
AWS additional steps