Skip to main content

Databricks

Visualize Databricks workspace groups, users, and clusters, map Databrick users to employees, and monitor changes through queries and alerts.

Installation​

To install this integration, you will need to configure settings both within Databricks and on JupiterOne. Before enabling in JupiterOne, ensure that you complete the setup within your Databricks account.

Configuration in Databricks​

The integration authenticates against a single Databricks workspace with a personal access token. Follow the steps for the cloud your workspace runs on to find the workspace URL, then generate the token.

Databricks on AWS​

  1. Go to the Databricks AWS account console and log in.
  2. In the Workspaces section, choose the workspace you want to ingest.
  3. The workspace URL has the format https://[deployment-name].cloud.databricks.com, for example https://dbc-c50dbe80-ed72.cloud.databricks.com. Take note of it and supply it on the integration configuration page.

Databricks on GCP​

  1. Go to the Databricks GCP dashboard and log in.
  2. In the Workspaces section, choose the workspace.
  3. You will be able to see the URL on the following page that has the following format: https://[numbers].[number].gcp.databricks.com
  4. Take note of it and supply it on the integration configuration page. For example, https://1122334455.6.gcp.databricks.com

Generating the access token​

  1. Click the workspace URL to go to the workspace dashboard. Once there, click on the settings icon (bottom part of left side menu) and choose User settings.
  2. Click Generate New Token, add comment/description, and press Generate. Retain this for use in JupiterOne.
info

For additional assistance generating an API Token on Databricks, see their documentation for more information.

Required Permissions in Databricks​

The Personal Access Token must have appropriate permissions to access the following resources:

  • Groups (/api/2.0/groups/list) - to retrieve group information
  • Group Members (/api/2.0/groups/list-members) - to retrieve user memberships in groups
  • Clusters (/api/2.0/clusters/list) - to retrieve cluster configurations

Important: The token user must have CAN_ATTACH_TO permission on clusters to successfully retrieve cluster information. Without this permission, the cluster list will return empty results. This permission can be granted through cluster access control lists (ACLs) in the Databricks workspace settings.

Optional: AWS configuration of a Databricks on AWS workspace​

The AWS resources backing a workspace — its region, VPC, subnets, security groups, root S3 bucket, KMS keys and PrivateLink settings — are only exposed through the Databricks account-level API, which a personal access token cannot call: "You can't use personal access tokens to automate Databricks account-level functionality."

To ingest them, additionally configure an account-level service principal with an OAuth secret:

  1. Go to the Databricks AWS account console and log in as an account admin.
  2. Copy your Account ID from the user menu in the top right corner.
  3. Go to User management > Service principals and either select an existing service principal or click Add service principal.
  4. On the service principal's Roles tab, enable Account admin. Account-level APIs require the service principal to be an account admin.
  5. On the Secrets tab, click Generate secret. Retain the Client ID and the Secret — the secret is only shown once.

The service principal reads the following account-level endpoints:

  • GET /api/2.0/accounts/{account_id}/workspaces - to find the account record of the configured workspace
  • GET /api/2.0/accounts/{account_id}/networks/{network_id} - to retrieve the VPC, subnets and security groups
  • GET /api/2.0/accounts/{account_id}/storage-configurations/{storage_configuration_id} - to retrieve the root S3 bucket
  • GET /api/2.0/accounts/{account_id}/credentials/{credentials_id} - to retrieve the cross-account IAM role, which identifies the AWS account
  • GET /api/2.0/accounts/{account_id}/customer-managed-keys/{customer_managed_key_id} - to retrieve the KMS keys
  • GET /api/2.0/accounts/{account_id}/private-access-settings/{private_access_settings_id} - to retrieve the PrivateLink settings
note

These fields are optional. When they are left empty, the integration ingests the workspace, its groups, users and clusters exactly as before. They have no effect on workspaces hosted on GCP or Azure.

Configuration in JupiterOne​

To install the Databricks integration in JupiterOne, navigate to the Integrations tab in JupiterOne and select Databricks. Click New Instance to begin configuring your integration.

Creating an instance requires the following:

  • The Account Name used to identify the Databricks account in JupiterOne. Ingested entities will have this value stored in tag.AccountName when the AccountName toggle is enabled.

  • Description to assist in identifying the integration instance, if desired.

  • Polling Interval that you feel is sufficient for your monitoring needs. You may leave this as DISABLED and manually execute the integration.

  • Your Databricks Host and Access token.

  • Optionally, for a Databricks on AWS workspace, your Databricks Account ID, the Service Principal Client ID and the Service Principal OAuth Secret of the account-level service principal described above.

Click Create once all values are provided to finalize the integration.

Next steps​

Now that your integration instance has been configured, it will begin running on the polling interval you provided, populating data within JupiterOne. Continue on to our Instance management guide to learn more about working with and editing integration instances.