Create an All-Purpose Cluster

You've just written your first notebook commands — but they would have run on Databricks' default cluster, costing you money and tuning flexibility. You need a cluster you own, configured for your workload, not someone else's. Creating and configuring an all-purpose cluster is the foundational skill you need to run ad-hoc analytics, develop code, and debug Spark jobs without waiting for a server to start — and without overpaying for resources you don't use. This lesson walks you through the exact steps, the configuration choices that matter, and the common pitfalls that trip up beginners.

The problem this lesson solves

Databricks gives you a multi-tenant compute layer, but a default cluster doesn't know if you need 4 GB of memory or 400. When you use a shared, unconfigured resource, you face three problems:

• Cost overruns — idle clusters burn Databricks Units (DBUs) every second they're up.
• Performance mismatch — a cluster too small for your data causes Spark tasks to spill to disk; one too big wastes money.
• Inconsistent environments — team members can't reproduce your results if every cluster uses different settings.

The solution is an all-purpose cluster, designed for interactive development — notebooks, jobs, and ad-hoc queries. Unlike job clusters, which Databricks spins up and automatically terminates, an all-purpose cluster stays running until you stop it, giving you an always-ready workspace. But that convenience comes with responsibility: if you forget to terminate it, it keeps costing you money. The goal of this lesson is to give you the skills to create, configure, and manage these clusters efficiently.

Core concept / mental model

Think of an all-purpose cluster as a rental car versus a leased car. A job cluster is a rental — you pick it up for a specific trip and return it immediately after. An all-purpose cluster is a lease — you keep it in your garage, ready for any errand, but you pay monthly even when it's parked.

In technical terms, an all-purpose cluster is a set of VMs (also called nodes) that Spark coordinates. Each cluster has:

• Driver node — the brain; it runs the Spark context and coordinates tasks.
• Worker nodes — the muscle; they execute the tasks and store data in memory or on disk.
• Cluster policies — a set of rules that restrict what configurations you can set, often enforced by administrators.
• Databricks Runtime — the version of Spark plus pre-installed libraries, such as Delta Lake and MLflow.

A key mental model is the cost equation: each node has a specific price per hour based on its VM type. The total cost is (driver cost + workers cost) * uptime. When you configure a cluster, you're balancing three variables:

1. Compute power — CPU cores and memory per node.
2. Scalability — how many nodes you can add as workloads grow.
3. Persistence — how long the cluster stays alive.

Once a cluster is created, it goes through phases: Pending → Running → Terminated. The autoscaling feature adjusts worker counts between a minimum and maximum you define, so you don't pay for idle capacity.

How it works step by step

Creating and configuring an all-purpose cluster follows a repeatable workflow. Here's the step-by-step process from the Databricks UI:

1. Navigate to the Compute pane

Click Compute in the left sidebar, then Create Compute. The screen shows a New all-purpose cluster form by default.

2. Name and configure the cluster

• Cluster name — use a descriptive name like data-eng-sanbox.
• Policy — choose a cluster policy (if any); newer workspaces enforce policies with preset defaults.
• Access mode — select Single user or No isolation shared (for Python only). Shared access with Scala requires additional setup.
• Databricks Runtime version — pick a Long-Term Support (LTS) version for stability; e.g., 13.3 LTS (Spark 3.4).
• Node type — select the VM instance type. Use Auto to let Databricks pick a balanced option, or choose manually based on memory and core needs.
• Minimum and maximum workers — set the autoscaling range. For development, 1 to 4 is a good starting point.
• Terminate after — set an idle timeout (in minutes) to avoid runaway costs. Recommended: 30 minutes for development.

3. Advanced options (optional but powerful)

In the Advanced options tab, you can:

• Set Spark config — pass custom Spark properties like spark.sql.shuffle.partitions.
• Add environment variables.
• Attach init scripts or libraries (e.g., install requests or a JDBC driver).
• Specify cluster log delivery for debugging.

4. Create and wait

Click Create Cluster. The cluster moves to Pending state, then Running once ready. You can attach notebooks by selecting the cluster from the dropdown in the notebook toolbar.

5. Stop when done

When you've finished, click Terminate to stop billing. The cluster's configuration stays saved so you can restart it later.

The whole process takes under 5 minutes, and the steps are identical whether you use the UI or the CLI/API — which is perfect for scripting.

Hands-on walkthrough

Let's put theory into practice. We'll create a small, cost-effective cluster and verify it works.

Exercise 1: Create a cluster in the UI

1. Go to Compute → Create Compute.
2. Name it my-dev-cluster.
3. Set Access mode to Single user.
4. Pick the latest LTS Databricks Runtime (e.g., 13.3 LTS).
5. Leave Node type as Auto.
6. Set Minimum workers to 1, Maximum workers to 2.
7. Under Advanced options, set Terminate after to 30 minutes.
8. Click Create Cluster.

Wait for the cluster to reach Running. Now attach a notebook and test:

# In a notebook attached to my-dev-cluster
from pyspark.sql import SparkSession

# Create a simple DataFrame
sdf = spark.range(1000).selectExpr("id", "id * 2 as doubled")
print(f"Partition count: {sdf.rdd.getNumPartitions()}")
sdf.show(5)

Expected output (trimmed):

Partition count: 2
+---+-------+
| id|doubled|
+---+-------+
|  0|      0|
|  1|      2|
|  2|      4|
|  3|      6|
|  4|      8|
+---+-------+

The partition count reflects the number of workers (2), confirming the cluster is operational.

Exercise 2: Use the Databricks CLI to create a cluster (optional)

If you prefer automation, the CLI lets you create a cluster from a JSON config. First, install and configure the CLI:

# Install the Databricks CLI (v0.0.19+)
pip install databricks-cli

# Configure (you'll need a personal access token)
databricks configure --host https://<your-workspace-url>

Then save the config as cluster.json:

{
  "cluster_name": "cli-dev-cluster",
  "spark_version": "13.3.x-scala2.12",
  "node_type_id": "i3.xlarge",
  "num_workers": 2,
  "autoscale": {
    "min_workers": 1,
    "max_workers": 4
  }
}

Finally, create the cluster:

databricks clusters create --json @cluster.json

The CLI returns a cluster ID. You can then list your clusters:

databricks clusters list

Exercise 3: Verify autoscaling

With autoscaling set to 1–4, load a larger dataset to see the scale-up:

# Trigger a shuffle-heavy operation
large_df = spark.range(10_000_000).repartition(8)
print("Partitions after repartition:", large_df.rdd.getNumPartitions())

Watch the cluster's Event log tab; you'll see workers being added as tasks launch.

Compare options / when to choose what

Not all clusters are equal. Here's a comparison of all-purpose vs. job clusters vs. serverless, and the main configuration choices.

Choosing node types

For most development work, Auto is the safest choice. As your workloads grow, switch to a high-memory node for shuffle-heavy operations.

Autoscaling strategy

• Development: set min=1, max=2 — that's enough for testing, cheap when idle.
• Production: min >= 4 for redundancy, max scaled to handle peak load.
• Static sizing: sometimes predictable loads benefit from fixed workers (no scale-up latency).

Troubleshooting & edge cases

Even an experienced user hits issues. Here are the most common ones with fixes:

Cluster fails to start (Cloud Provider Failure)

Symptom: the cluster stays in Pending and then shows an error like Cloud provider failure. Causes and fixes:

• Quota exceeded — you've reached your cloud account's vCPU or instance limit. Contact your cloud admin or reduce the worker count.
• Spot instance unavailability — if using spot instances (marked as * in the UI), the provider can't allocate instances. Switch to on-demand for critical work.
• VPC/network misconfiguration — in advanced setups, security groups may block traffic. Check the cluster's Driver logs or event history for the exact error.

Notebook runs slowly

Symptom: your query is slow even with many workers. Check:

• Data skew — some partitions have far more data than others; repartition or use salting.
• Small cluster for big data — increase the max workers or choose high-memory nodes.
• Autoscaling lag — initial scale-up takes minutes; for interactive work, set a higher minimum.

Cost surprise at the end of the month

Symptom: your bill is higher than expected. Fix:

• Always set an idle timeout — even 30 minutes saves money.
• Terminate clusters manually after long idle periods.
• Use cluster pools (if available) to reduce startup wait, but remember pools themselves cost money when idle.

Runtime version mismatch

Symptom: a library not installed in your current runtime. Fix: choose a runtime that includes the library (check the Databricks runtime release notes) or install it via Libraries tab on the cluster.

Pro tip: Use the Event log tab to see why a cluster was terminated. It's the first place to look when debugging cluster lifecycle issues.

What you learned & what's next

In this lesson, you've learned how to create and configure an all-purpose cluster — the core compute unit for Databricks development. You now know how to:

• Explain the role of an all-purpose cluster vs. job clusters.
• Navigate the Compute UI and create a cluster with custom settings.
• Configure autoscaling, node types, and idle timeout to balance cost and performance.
• Use the CLI to automate cluster creation.
• Troubleshoot common cluster startup and performance issues.

You've also completed a hands-on exercise that proves your cluster can run Spark code — a practical skill you'll use every day.

Next in the track: Configure cluster policies and manage libraries. These lessons allow you to share pre-approved configurations across teams and install dependencies at scale. With your cluster ready, you're set to explore them.