Run Your First Databricks Notebook

You've heard the hype: Databricks is the unified analytics platform that combines data lakes and warehouses into a single Lakehouse. But all the talk about clusters, notebooks, and Spark jobs can feel abstract until you actually run your first notebook. This lesson cuts through the noise — you'll go from zero to a running notebook on Databricks, covering workspace creation, cluster setup, and notebook execution, so you can see results (and feel the rush) before you dive deeper.

The Problem This Lesson Solves

Starting with Databricks is overwhelming. The platform has a console, clusters, notebooks, jobs, and SQL endpoints — where do you even begin? Many beginners spend hours clicking around, creating a cluster that costs money, or wondering why their notebook cells don't execute. Without a guided path, you risk:

• Wasting credits on idle clusters.
• Getting lost in the UI's many menus and options.
• Frustration from unclear error messages.

This lesson gives you a clear, efficient path: create a workspace, spin up a cluster (with the right settings), and run your first notebook. You'll learn the minimal steps, not every button in the console. By the end, you'll have a working environment and confidence to explore further.

Core Concept / Mental Model

Think of Databricks as a data kitchen where:

• Workspace is the kitchen counter where you prepare your work. It holds your notebooks, libraries, and results.
• Cluster is the stove and oven — the computation that heats your data. Without it, you can write recipes, but you can't cook.
• Notebook is your recipe card. It contains code cells that you can run independently, and each cell prints its own status (like 'DONE' or errors).
• Spark contexts are the pots and pans — they connect your notebook to the cluster's processing engines.

When you run a notebook cell, Databricks sends the code to the cluster's Spark engine, which executes it and returns the result to your notebook. The notebook does not run locally; it's an interactive interface to the cluster.

Key terms to know: - Workspace: Your project folder structure, like a file explorer. - Cluster: A set of virtual machines (nodes) that processes data. - Notebook: A document with code cells, markdown, and visualizations. - Cell: One executable block of code.

Pro tip: Think of your cluster as a shared oven. If you're not cooking, turn it off to save money! Databricks charges per-second for running clusters.

How It Works Step by Step

Here's the high-level workflow for running your first notebook:

1. Sign up for a Databricks account (Community Edition is free for testing).
2. Create a workspace — your project area in the cloud.
3. Launch a cluster — with a small configuration (e.g., 2 cores, 8 GB).
4. Create a notebook — attach it to the cluster.
5. Write and run cells — Spark or SQL code that processes data.
6. View results — in the output area below the cell.

Each step has a specific purpose and builds on the previous. Let's walk through each in detail.

Hands-On Walkthrough

Step 1: Create Your Workspace

If you're using Community Edition:

1. Go to databricks.com/try and sign up.
2. Choose the Community Edition (free) — it's enough for learning.
3. After email verification, log in. You'll see the Databricks workspace UI.

For a cloud trial (AWS, Azure, GCP), follow the provider's sign-up steps — they'll provision a workspace for you.

Step 2: Start a Cluster

From the left sidebar, click Compute (or Clusters). Then:

1. Click Create Cluster.
2. Give it a name like learning-cluster.
3. Choose the runtime version (e.g., 12.2 LTS — latest stable).
4. For Community Edition, keep default settings (single node).
5. Click Create Cluster. It will take 1–2 minutes to start.

Pro tip: Use the Auto Terminate setting (default 120 minutes) to avoid spending money when you forget to turn it off.

Step 3: Create a Notebook

From the sidebar, click Workspace:

1. Click the Create button (top-right) and select Notebook.
2. Name it first-notebook.
3. Choose Python as the default language.
4. Make sure your cluster is attached (dropdown at top-right shows your cluster name).
5. Click Create. An empty notebook opens with one cell.

Step 4: Write and Run Your First Cell

Let's start with a classic — print a message and run a simple Spark operation.

# Cell 1: Hello, Databricks!
print("Hello, Databricks!")

# And a simple Spark DataFrame
from pyspark.sql import SparkSession

# SparkSession is already available as 'spark' in Databricks
# Create a list of data
sample_data = [("Alice", 34), ("Bob", 45), ("Cathy", 29)]

# Convert to DataFrame
spark_df = spark.createDataFrame(sample_data, ["name", "age"])

# Show the data
spark_df.show()

Expected output:

Hello, Databricks!
+-----+---+
| name|age|
+-----+---+
|Alice| 34|
|  Bob| 45|
|Cathy| 29|
+-----+---+

Run the cell by clicking the Run button (▶) in the cell's right side, or press Shift+Enter. You'll see the cell status become 'RUNNING' and then 'DONE' with the output below.

Step 5: Try a SQL Cell

You can mix languages in one notebook. In a new cell, switch to SQL using the language dropdown (next to the cell's run button). Then paste:

-- Cell 2: SQL query on a table we just created
-- First, register the DataFrame as a temporary view
-- (in Python cell) spark_df.createOrReplaceTempView("people")

SELECT * FROM people WHERE age > 30;

But wait — you need to register the view first. Go back to Cell 1 (Python), add spark_df.createOrReplaceTempView("people") at the end, and run it. Then run the SQL cell.

Expected output:

alice 34
bob 45

Pro tip: Use display() instead of show() for interactive visualizations. Try display(spark_df) in a Python cell.

Step 6: Bonus — Visualize Your Data

Run a cell with display(spark_df) — you'll get an interactive table and chart options. It's a powerful way to explore data without leaving the notebook.

Compare Options / When to Choose What

Notebook options: - Python, Scala, SQL, R — you can switch languages per cell. Python is easiest for beginners. - Single-node vs multi-node clusters — for learning, single-node is fine; for big data, use multi-node.

Pro tip: If you're just learning, Community Edition is perfect. When you need real data, move to a cloud trial.

Troubleshooting & Edge Cases

Cluster not starting?

• Error: "Cluster terminated" or "Insufficient capacity."
• Fix: Wait a few minutes; try again. For Community Edition, resources are shared — if it fails, delete and recreate.
• Check: Ensure runtime is LTS — older versions may have issues.

Notebook cell status stuck on 'RUNNING'?

• Cause: Cluster is still starting, or your code is heavy.
• Fix: Wait until cluster is Running. Then run again. If stuck forever, restart the Python kernel via Kernel > Restart.

"No module named 'pyspark'"?

• Cause: You're trying to run Spark on a notebook not attached to a cluster.
• Fix: In the notebook toolbar, select your cluster from the dropdown. If none, create one (Compute > Create Cluster).

SQL cell returns 'Table not found'?

• Cause: You didn't create the temporary view first.
• Fix: Run the Python cell that creates the view before the SQL cell. In a fresh notebook, run cells in order.

Accidentally deleted a cell or notebook?

• Fix: Databricks has a version history. Click File > Version History to restore.

What You Learned & What's Next

Congratulations! You've successfully run your first Databricks notebook. To recap:

• You can now create a workspace (Community Edition or cloud) and launch a cluster.
• You can create a notebook, attach it to a cluster, and run Python and SQL cells.
• You understand the mental model: workspace (folder), cluster (computation), notebook (code script).
• You've practiced with a hands-on exercise — printing messages, creating a DataFrame, and running a SQL query.
• You know how to troubleshoot common issues like cluster startup failure, stuck cells, and missing modules.

This foundation sets you up for the next lesson: Incorporating Data with Databricks — where you'll learn to read data from files, create Delta tables, and prepare your first ETL pipeline. You'll build on the notebook and cluster you just mastered.

Final tip: Practice by creating a new notebook and exploring the Spark DataFrame API. Try df.select(), df.filter(), or even df.groupBy(). The more you experiment, the more comfortable you'll become. Happy data engineering!