Databricks CLI Secrets Management

You've spent weeks perfecting your Databricks notebooks and pipelines, and they're running beautifully—until a hardcoded database password leaks into a Git repo and your security team sends you a very unpleasant message. Hardcoding secrets in code or notebooks is a fast track to credential leakage, compliance violations, and painful rotation. This lesson teaches you how to handle secrets with the Databricks CLI: a command-line approach to securely store, manage, and reference credentials so your code stays clean and your secrets stay safe.

The problem this lesson solves

Every production data pipeline relies on credentials: database usernames, API tokens, cloud service keys, and more. The natural instinct is to embed them directly in your notebooks or Python scripts—quick, convenient, and absolutely dangerous. Here's why:

• Version control leaks: If your repo is ever shared or made public, every stored secret goes with it.
• Access control gaps: Anyone with read access to the notebook sees the credentials, even if they shouldn't.
• Rotation nightmares: When a credential must be revoked, you have to hunt down every instance and update it manually.
• Audit trail missing: No record of who accessed what, when, or why.

The Databricks CLI provides a secrets management layer that lets you store sensitive values in a centralized, encrypted store and reference them in code via dbutils.secrets. This keeps credentials out of your source code and gives you fine-grained control over who can access which secret. By the end of this lesson, you'll be able to create secrets, reference them securely, and avoid the trap of hardcoding.

Core concept / mental model

Think of the Databricks CLI secrets feature as a secure key-value vault layered on top of your Databricks workspace. You create a scope (a named container), store secret keys inside it, and then grant specific users or service principals permission to read those keys. Your notebooks and jobs only ever see the key, never the secret value.

Here's a mental diagram:

• Scope — like a folder or namespace for secrets (e.g., my_app_secrets).
• Secret Key — a name inside the scope (e.g., db_password).
• Secret Value — the actual credential (e.g., S3cr3tP@ss).
• Permissions — who is allowed to read that key.

In your code, you use dbutils.secrets.get(scope='my_scope', key='db_password') to retrieve the credential at runtime. The CLI only helps you manage the store—it never writes plaintext secrets into your scripts.

Pro tip: The Databricks CLI secrets commands are a thin wrapper around the Secrets REST API. They perform the same operations you'd do with curl but pack them into a friendlier interface.

Key terms you'll see

• Databricks CLI — a command-line tool that talks to Databricks APIs.
• Secrets Scope — the top-level namespace for secrets.
• Backend — where the scope is stored: either Databricks-managed or Azure Key Vault-backed.
• dbutils.secrets — the utility available in notebooks and jobs to read secrets.

How it works step by step

Here's the logical flow to handle secrets with the Databricks CLI:

1. Install or update the Databricks CLI — ensure you have a recent version that supports secrets.
2. Authenticate — configure the CLI with your workspace URL and a personal access token.
3. Create a secrets scope — pick a name and choose a backend (databricks or azure-keyvault).
4. Add secret values — store each credential as a key-value pair inside the scope.
5. Grant access — allow specific users or groups to read the secrets.
6. Reference secrets in code — use dbutils.secrets.get() in notebooks or jobs to fetch values at runtime.
7. Rotate and manage — update or delete secrets as needed without touching your code.

Each step is a CLI command or a small snippet, so let's see them in action.

Hands-on walkthrough

Step 1: Prepare your environment

First, confirm the CLI is installed and authenticated. If you haven't done this, run:

# Install the Databricks CLI (macOS/Linux)
pip install databricks-cli

# Configure authentication
databricks configure --host https://your-workspace.cloud.databricks.com
databricks configure --token

You'll be prompted to enter a personal access token. For automation, you can also set environment variables like DATABRICKS_HOST and DATABRICKS_TOKEN.

Step 2: Create a secrets scope

# Create a Databricks-managed secrets scope named 'my_app_secrets'
databricks secrets create-scope --scope my_app_secrets --backend databricks

Expected output (nothing printed on success) — verify with:

databricks secrets list-scopes

You should see my_app_secrets listed.

Pro tip: If you're using Azure, you can back the scope with Azure Key Vault by passing --backend azure-keyvault and --resource-id <key-vault-uri>. That way Databricks stores a pointer to your existing vault instead of holding secret values itself.

Step 3: Store a secret

# Store a database password under the key 'db_password'
databricks secrets put --scope my_app_secrets --key db_password --string-value "S3cr3tP@ss"

Using --string-value passes the value directly. For non-interactive environments, you can pipe it in:

echo "S3cr3tP@ss" | databricks secrets put --scope my_app_secrets --key db_password

Expected behavior: The CLI writes the secret to the backend. You'll see no output on success.

Verify the secret exists (but won't show its value):

databricks secrets list --scope my_app_secrets

You'll see the key name db_password — not the value.

Step 4: Grant access (Databricks-managed scopes only)

For databricks backend scopes, you need to grant read permission to users or groups. For the users group (which includes all users), run:

databricks secrets add-acl --scope my_app_secrets --principal users --permission MANAGE

Permission levels: READ, WRITE, MANAGE. READ lets a principal read secret values; WRITE allows adding/updating secrets; MANAGE gives full control over the scope and its ACLs.

Note: If you use the azure-keyvault backend, ACLs are managed in Azure Key Vault itself—Databricks does not handle permissions there.

Step 5: Reference the secret in your notebook or job

Now in a Python notebook, you can write:

from pyspark.sql import SparkSession

# Fetch the stored secret value
password = dbutils.secrets.get(scope="my_app_secrets", key="db_password")

# Use it to connect to a database (e.g., JDBC)
jdbc_url = "jdbc:postgresql://my-db-host.example.com:5432/mydb"
df = (
    spark.read
    .format("jdbc")
    .option("url", jdbc_url)
    .option("dbtable", "public.customers")
    .option("user", "etl_user")
    .option("password", password)
    .load()
)

df.show(5)

The secret value is resolved at runtime and never appears in plaintext in your notebook or source code.

Step 6: Rotate a secret

When a credential expires, just overwrite it:

databricks secrets put --scope my_app_secrets --key db_password --string-value "NewSecret456"

No code changes needed—the next run picks up the new value automatically. To remove a secret completely:

databricks secrets delete --scope my_app_secrets --key db_password

Compare options / when to choose what

When you're just getting started, the managed backend is easiest. If you already use Azure Key Vault or need centralized secret management across multiple cloud services, choose the Key Vault backend.

Variations to consider

• Using environment variables — For local development, you might export secrets as env vars and read them in code with os.environ. But this scatters secrets across your environment and doesn't scale for team collaboration.
• Secret scopes via REST API — If you're writing automation, you can skip the CLI and call the REST API directly. It's useful for CI/CD pipelines where you can't install the CLI.
• Terraform Provider for Databricks — For infrastructure-as-code, the provider manages secret scopes and ACLs, integrating versioned secret definitions into your IaC workflows.

Troubleshooting & edge cases

• Error: Failed to fetch secrets — Usually means your authentication token lacks the right permission. Make sure you're authenticated (databricks configure) and your token has SECRETS_CREATE or SECRETS_READ permissions on the scope.
• Secret key not found — You may be referencing a scope/key that doesn't exist. Double-check the scope name and key with databricks secrets list-scopes and databricks secrets list --scope <name>.
• Permission denied in notebook — Your user or service principal hasn't been granted READ on the scope. Run databricks secrets add-acl --scope my_app_secrets --principal <user> --permission READ.
• Secrets visible in logs — Never print secret values to logs; use them only where needed. If you accidentally do, rotate the secret immediately.
• Workspace-scoped vs. global scopes — Databricks scopes are tied to a workspace. If you have multiple workspaces, you'll need to recreate scopes in each one unless you use Key Vault backend.

What you learned & what's next

This lesson showed you how to handle secrets with the Databricks CLI: from storing credentials in secure scopes to retrieving them in code via dbutils.secrets. You learned the mental model of scopes, keys, and ACLs, and practiced the full workflow—create, store, grant, reference, and rotate. You can now explain the core idea behind secret management with the Databricks CLI and complete a practical exercise, meeting both learning objectives.

Next step: Move on to lesson 22: Managing clusters with Databricks CLI. You'll use your secrets in cluster configurations—for example, storing DB credentials for Spark JDBC. Secure secret handling will give you a solid foundation for automating infrastructure and pipelines next.