AWS Databricks Setup

Some companies require different cloud devops teams to be involved with this step for security reasons. Please contact [e-mail] for any assistance.

Step 1: Cloud provider and region

• Cloud Provider - Select AWS as your cloud provider

• AWS Region - This is necessary only if you specify AWS as your cloud provider.

• Logs and Data Collection - Choose how you want to provide logs to Gradient. We recommend Sync-Hosted collection which manages collecting logs with just a few configurations on your end. Self-Hosted requires additional set up on your end.

• Monitoring Type - Choose how you want Gradient to monitor your Databricks clusters.

  • We recommend EventBridge Rule monitoring which sets up your AWS environment to send EC2 related events to Gradient. Only available if you pick Sync-Hosted for Logs and Data Collection. More information on this page.

  • Another Monitoring Type is Webhook which is covered on this page.

For the recommended Sync-Hosted collection method, AWS IAM roles and permissions are required to be set up to complete the rest of the workspace integration, as seen in the screen shot below.

We recommend using Sync-Hosted collection for your logs and other data. Sync-Hosted collection enables you to integrate with Gradient in 5 minutes.

Monitoring Type EventBridge Rule is currently available under Private Preview.

We recommend using EventBridge Rule monitoring for monitoring your Databricks clusters. Only available if you also pick Sync-Hosted collection.

Step 2: Create AWS IAM Permissions Policy

These steps need to be performed in the AWS Account that is associated with the Databricks Workspace. The screenshot below shows you how to locate the correct AWS Account.

Copy and paste the json in a new AWS IAM policy permission, as see in the example screen shot below. Give the policy a name, such as sync-external-access

This permission will be used in the next few steps after a new role is created.

Are your Databricks cluster logs stored in S3?

If so, we'll need permissions to fetch cluster logs from your S3 bucket.

Use the policy below to grant Sync access to your S3 location where cluster logs are stored. Replace the bucket names and prefixes with yours.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeInstances",
                "ec2:DescribeVolumes"
            ],
            "Resource": "*"
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::your-bucket-and-prefix1-here/*",
                "arn:aws:s3:::your-bucket-and-prefix2-here/*"
            ]
        }
    ]
}

Are your Databricks cluster logs stored in DBFS?

If your cluster logs are stored in DBFS then the policy below will suffice.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeInstances",
                "ec2:DescribeVolumes"
            ],
            "Resource": "*"
        }
    ]
}

AWS IAM Policy: sync-external-access

Step 3: Create an AWS IAM Role (custom trust policy)

Create a new AWS IAM role with the "Custom trust policy" trust entity and paste the JSON in the policy field, as seen in the example screenshot below. Give the role a name, such as sync-external-user-role

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::533267411813:role/sync-computing-collector"
            },
            "Action": "sts:AssumeRole",
            "Condition": {
                "StringEquals": {
                    "sts:ExternalId": "<externalId from Gradient>"
                }
            }
        }
    ]
}

AWS IAM Custom Trust Role: sync-external-user-role

Step 4: Add permission to role

In the next step of the AWS IAM role creation, add the permission created previously to the new AWS IAM role (in the example above it is named sync-external-access). Example screen shot below:

Step 5: Name and create the IAM role

Give the new IAM role a name, such as sync-external-user-role, and create the new role.

Step 6: Copy and paste the AWS IAM Role ARN

Go back to the AWS IAM role just created (in the previous example with the name sync-external-user-role), and copy the ARN link and paste it in the last field in the Gradient dialogue box.

Once entered, click on "Save & Test Access" to proceed in the Gradient UI.

Access check test - A series of access checks will occur to verify the permissions have been properly set up. If all checks are passed with green checks, you may proceed to the next section. If there are any red checks, please verify the installation is correct. Please email [e-mail] if there are any issues or questions.

Finally

• If you picked EventBridge Rule monitoring continue with the setup on this page:

EventBridge Setup

• If you picked Webhook monitoring continue with the setup on this page:

Webhook Setup