Manage Databricks Libraries

You've just written a brilliant PySpark transformation, hit Shift+Enter, and your notebook explodes with ModuleNotFoundError: No module named 'dateutil'. Your cluster is healthy, your code is logically sound, but your environment is missing a package. This is the reality of working on a managed Spark platform: your code is only as powerful as the libraries your cluster can see. In this lesson, you'll stop guessing and start managing libraries in Databricks—installing Python packages from PyPI and other sources, choosing the right installation scope, and eliminating environment-related failures for good.

The problem this lesson solves

Every Databricks cluster ships with a baseline set of Python packages (like pyspark, pandas, and numpy), but that baseline is never enough for real-world work. You'll inevitably need scikit-learn for a quick model, boto3 to read from S3, or a niche library like fuzzywuzzy for string matching. Without a deliberate approach to managing libraries, you face two painful extremes:

• Manual pip install in a notebook cell: Works until the cluster restarts. Then your environment resets, and every new session starts with a search-and-install ritual.
• Installing libraries globally: Wastes cluster resources and creates version conflicts between projects sharing the same cluster.

The deeper problem is that your dependencies are often undocumented. When a teammate attaches to your cluster and your notebook fails with an import error, you've lost time and trust. In this lesson, you'll learn how libraries behave inside Databricks, how to install them at the right scope, and how to make your environments reproducible.

By the end, you'll be able to explain why you chose a cluster library over a notebook-scoped library, and you'll have a step-by-step mental model for handling any package installation—from PyPI, Maven, or even a custom JAR.

Core concept / mental model

Think of library management in Databricks like provisioning a toolbox for a shared workshop. The cluster is the workshop itself—it has a default set of tools (the runtime's built-in libraries) that every worker can use. When you install a cluster library, you're adding a drill to the communal bench: every notebook attached to that cluster can grab it, and it stays there until you explicitly remove it.

A notebook-scoped library is like borrowing a specialized torque wrench for a single job. You install it from within the notebook, it works for that session, and when the session ends, the wrench returns to the drawer—nothing persistent, no impact on your teammates.

Libraries in Databricks come in several flavors, and each has a natural use case:

Definitions to remember

• Cluster library: A library installed at the cluster level. It's available to all notebooks and persists across sessions while the cluster runs. If the cluster is resized or restarted, the library is reinstalled automatically (because it's part of the cluster definition).
• Notebook-scoped library: Installed using %pip or dbutils.library.install inside a notebook. It's ephemeral—gone when the cluster restarts or the notebook detaches.
• Init scripts: A shell script that runs on cluster startup. You can use it to install packages or configure environments before your code runs.

The key insight: Library management is about scope and reproducibility. You install once at the cluster level for shared, stable dependencies, or install in a notebook for quick, disposable experiments. Never rely on pip install inside a notebook cell for anything you need tomorrow.

How it works step by step

Follow this decision framework whenever you need a new package:

1. Identify the exact package name and version you need. For Python, check PyPI. For Spark connectors, check Maven Central.
2. Decide the scope: Will multiple notebooks on this cluster use it? If yes → cluster library. If it's a one-off for a single notebook → notebook-scoped.
3. Install using the appropriate method: - Cluster library via UI or databricks-cli/Terraform for automation. - Notebook-scoped using %pip install.
4. Verify the installation by importing the package in a fresh cell.
5. Handle transitive dependencies — Databricks automatically resolves them for PyPI libraries, but for custom JARs you might need to add them manually.
6. Document your libraries — either in the cluster's library list or in a requirements file attached to your repo.

How cluster library installation works under the hood

When you attach a PyPI library to a cluster, Databricks does the following: - During cluster startup (or on the next restart), it runs pip install on every node (driver and workers) with the package name and version you specified. - It records the library as part of the cluster's definition, so any auto-restart or resize will reinstall it automatically. - The library becomes visible to all notebooks attached to the cluster.

For Maven libraries, Databricks resolves the artifacts from Maven Central or a custom repository, downloads the JARs, and adds them to Spark's classpath.

Notebook-scoped installs: %pip

Databricks launched %pip to make in-notebook installs first-class. It's a magic command that behaves like a pip command but is scoped to the notebook's session. Under the hood, it:

• Installs the package on the driver and all workers (since the notebook runs on the Spark driver and executors).
• Makes the package available immediately—no cluster restart needed.
• Does not persist after the notebook is detached or the cluster restarts.

Init scripts: the ultimate fallback

If you need custom shell commands during startup (e.g., install a specific version of a system library like libpq-dev), init scripts are the answer. They run on every node before the Spark interpreter starts. This is the most flexible but also the most error-prone method—use it sparingly.

Hands-on walkthrough

Let's walk through both core methods with real examples.

Method 1: Install a notebook-scoped library with %pip

Use this for quick experiments. For example, install the requests library (which is actually bundled, but let's simulate) or a library that's not included, like holidays.

# In a Databricks notebook cell, use the %pip magic command
%pip install holidays==0.29

Expected output:

Collecting holidays==0.29
  Downloading holidays-0.29-py3-none-any.whl (527 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 527.0/527.0 kB 1.2 MB/s eta 0:00:00
Installing collected packages: holidays
Successfully installed holidays-0.29

Now use the library in the same notebook:

import holidays

us_holidays = holidays.US(years=2024)
print("New Year's Day:", us_holidays.get('2024-01-01'))

Expected output:

New Year's Day: New Year's Day

The package is available in all subsequent cells of the session.

Method 2: Attach a cluster library via the UI

For a library your whole team needs, do this:

1. Go to your Compute page (Compute → select your cluster).
2. Click the Libraries tab.
3. Click Install New → select PyPI.
4. Enter scikit-learn and pin a version like 1.5.1 (avoid floating versions for reproducibility).
5. Click Install.
6. Wait for the status to change from Pending to Installed.
7. Test it from a notebook attached to this cluster:

from sklearn.ensemble import RandomForestClassifier

# This will raise an error if the library isn't properly installed
print(RandomForestClassifier())

Expected output:

RandomForestClassifier()

Pro tip: If a library fails to install, check the error log under the library's status. A common issue is a non-existent version—always verify the version on PyPI first.

Automating cluster libraries with the Databricks CLI

For reproducibility, define libraries as code. Here's an example using the Databricks CLI (assuming you've configured it):

# Create a JSON file describing the library
cat > library.json <<'EOF'
{
  "libraries": [
    {
      "pypi": {
        "package": "scikit-learn==1.5.1"
      }
    }
  ]
}
EOF

# Update an existing cluster (replace CLUSTER_ID)
databricks clusters update --json '{"cluster_id": "1234-567890-cluster123", "libraries": [{"pypi": {"package": "scikit-learn==1.5.1"}}]}'

This approach makes your cluster's dependency list auditable and versionable.

A complete, reproducible example: requirements + install

Create a requirements.txt file in your repo, then install all packages in one shot in a notebook:

# Cell 1: Show the requirements file (for clarity)
# In practice, you'd read it from DBFS or Git
requirements = ["holidays==0.29", "requests==2.32.3", "pyjanitor==0.27.0"]

# Cell 2: Install them all
%pip install -r requirements.txt

Remember: Installing via %pip with -r works only if the file is reachable from the notebook's filesystem. For a truly reproducible environment, attach a wheelhouse or conda environment file to a cluster.

Compare options / when to choose what

Here's a practical decision table for choosing between the main installation methods:

When to choose what: - Cluster libraries for anything your team or project consistently needs—data science libraries, connectors, etc. - Notebook-scoped for exploring a new package you might discard, or for a single analysis that doesn't need to be reproducible. - Init scripts only when you need binaries or system packages that pip can't provide (e.g., libgomp for certain ML libraries).

Variations to know

• Maven libraries: For Spark add-ons like com.databricks:spark-xml. Install via the UI, selecting Maven and entering the coordinates.
• Conda environments: You can attach a pre-built conda environment to a cluster using %conda or cluster-scoped conda base. This gives stronger dependency resolution than pip for some data science stacks.
• Custom wheel files: For proprietary or internal packages, upload a .whl file to DBFS or use a second package repository.

Troubleshooting & edge cases

ModuleNotFoundError after installing with %pip

• Cause: You installed the library in an earlier cell but the notebook was detached and reattached to the cluster—the session state was lost.
• Fix: Re-run the %pip install command. To avoid this, install it as a cluster library if you need it repeatedly.

Library shows as Installed but fails to import

• Cause: Version conflict with another package (e.g., numpy 2.x vs pandas 1.5).
• Fix: Check the cluster logs, then explicitly pin a compatible version. For example, if scikit-learn requires numpy<2, install numpy==1.26. Use %pip uninstall or remove the cluster library and re-add with pinned versions.

%pip says Error: failed to install with no further detail

• Cause: Usually a network or permission issue on your workspace (e.g., private VNet without internet access).
• Fix: Use a cluster library with a known-good PyPI mirror, or configure the cluster's spark.databricks.pyspark.sso.enabled and spark.databricks.egress.denyScala settings. If you're in a secure environment, ask your admin for a whitelist of package repositories.

Library disappears after cluster restart

• Cause: You used %pip—notebook-scoped libraries do not persist.
• Fix: Reinstall, or better, attach the package as a cluster library via the UI or infrastructure as code.

Wrong Python environment

• Cause: Databricks Runtime has multiple Python environments (e.g., the system Python vs. the Spark session's Python). %pip targets the current environment, but if you accidentally pip install via a subprocess, it might hit the wrong one.
• Fix: Always use %pip or the cluster library UI, never raw !pip install in a notebook.

What you learned & what's next

You can now manage libraries and install Python packages in Databricks with confidence. Specifically, you can:

• Explain the difference between cluster-scoped and notebook-scoped libraries and choose between them.
• Install packages from PyPI via the UI, %pip, or the CLI.
• Troubleshoot common installation failures and avoid environment drift.
• Apply best practices like pinning versions and documenting dependencies.

Common mistakes to avoid: - Using !pip install in a notebook cell instead of %pip—it often targets the wrong Python and doesn't work on the executors. - Installing packages without version pins, leading to non-reproducible results weeks later. - Ignoring the cluster's existing libraries and causing conflicts (e.g., upgrading numpy globally and breaking pandas). - Forgetting to reinstall notebook-scoped libraries after a cluster restart.

Next step: Your environment is predictable now. The next lesson in this track will show you how to schedule your notebooks as production jobs—putting your well-provisioned clusters to work on a regular cadence. You'll tie together your library strategy with job clusters, so every run starts with exactly the right dependencies.