Use Parameters to Parameterize Notebooks

You've just written a notebook that processes sales data perfectly — for one region, on one date. Then someone asks for the same report for another region. You copy the notebook, change a few hard-coded values, and hope you remember to fix the other three places. This is the pain that parameterization solves: without a way to inject inputs, every variation of your logic becomes a separate, brittle copy that rots over time. In this lesson, you'll learn how to make any Databricks notebook reusable by parameterizing it — so the same code runs for any input, in any environment, with zero copy-paste.

The problem this lesson solves

Hard-coded values are the silent killer of maintainable pipelines. Imagine a notebook that loads a Delta table, filters on 2024-01-01, and writes results to a path like /mnt/reports/sales_2024_01_01. It works — until another team asks for the same logic with a different date. Now you have two notebooks, and the next change requires editing both. Multiply that by ten regions, five date ranges, or three output formats, and you've created a maintenance nightmare.

Parameterization solves this by turning your notebook into a function — a reusable unit that accepts inputs and produces outputs. Instead of writing a new notebook for each variation, you write one notebook that reads its inputs from parameters. This reduces duplication, centralizes logic, and makes your pipelines auditable and testable.

Why care now? In any serious data engineering workflow, you'll soon need to run the same transformation with different inputs — as part of an orchestrated job, from a dashboard, or in a multi-tenant environment. Notebook parameterization is the foundational technique that makes your code reusable and your pipelines reproducible.

Core concept / mental model

Think of a parameterized notebook as a vending machine. The machine has a fixed interior — the logic that processes whatever you put in. You press a button (pass a parameter) to choose what you want. The machine runs the same internal steps, but the output changes based on your choice. In Databricks, widgets are the buttons, and dbutils.widgets.get() is the slot that delivers your input.

Two primary mechanisms exist for parameterization in Databricks:

• Notebook widgets: The built-in UI controls (dropdowns, text boxes) that let users set values interactively and let jobs pass values at runtime.
• dbutils.notebook.run(): The way to call another notebook from within a notebook, passing a dictionary of parameters.

Here's a mental diagram:

[Job / Orchestrator]  →  [Notebook with widgets]  →  (Logic)  →  [Output table/path]
        |                     ↑
        └─────── parameters ──┘

The notebook reads parameters at the top, uses them throughout, and produces results that depend on those inputs. The logic is the same; only the inputs change.

Pro tip: Treat your parameter names as a contract. If you change a widget name, every caller that passes that parameter will silently fail — use consistent naming like date_from, region, output_table.

How it works step by step

Here's the flow from creating a widget to using it in your notebook:

1. Create a widget using dbutils.widgets.text(name, default_value, label) — or dropdown, combobox, etc.
2. Retrieve the value using dbutils.widgets.get("name") — this returns a string, even if you expect an integer or date.
3. Type-cast and validate the value (e.g., convert to int, date, or Decimal).
4. Use the value in your Spark SQL, DataFrame transformations, or paths — after interpolation.
5. Optionally remove widgets after use with dbutils.widgets.removeAll() to keep the notebook clean when run as a job.

When you run the notebook from a job, the job submits parameters that override the widget defaults. The same notebook can be called with different parameters without any code changes.

Hands-on walkthrough

Let's build a reusable notebook that filters and aggregates data based on parameters.

Example 1: Basic parameterized filter

from pyspark.sql import functions as F
from datetime import datetime

# 1. Create widgets with defaults
dbutils.widgets.text("date_from", "2024-01-01")
dbutils.widgets.text("date_to", "2024-12-31")
dbutils.widgets.text("table_name", "sales")

# 2. Retrieve values (always strings!)
date_from = dbutils.widgets.get("date_from")
date_to = dbutils.widgets.get("date_to")
table_name = dbutils.widgets.get("table_name")

# 3. Convert to proper types
from datetime import datetime
date_from_dt = datetime.strptime(date_from, "%Y-%m-%d")
date_to_dt = datetime.strptime(date_to, "%Y-%m-%d")

# 4. Use the values in a query
filtered_df = spark.sql(f"""
  SELECT * FROM {table_name}
  WHERE date BETWEEN '{date_from}' AND '{date_to}'
""")

# 5. Show the count as output
print(f"Rows between {date_from} and {date_to}: {filtered_df.count()}")
filtered_df.display()

Expected output (when defaults are used):

Rows between 2024-01-01 and 2024-12-31: 12345

Example 2: Using dbutils.notebook.run() to call a parameterized notebook

Often you'll orchestrate multiple notebooks. Here's how one notebook can call another, passing parameters.

child_notebook (a parameterized notebook):

dbutils.widgets.text("region", "all")
dbutils.widgets.text("date", "2024-01-01")

region = dbutils.widgets.get("region")
date = dbutils.widgets.get("date")

print(f"Processing data for region={region} on date={date}")
# ... transformation logic ...

parent_notebook:

# Call the child notebook with custom parameters
result = dbutils.notebook.run("child_notebook", timeout_seconds=300, arguments={"region": "west", "date": "2024-06-01"})
print(result)

Expected output in the parent notebook:

Processing data for region=west on date=2024-06-01

Example 3: Parameterizing output paths (sink)

dbutils.widgets.text("output_table", "default_sales")
output_table = dbutils.widgets.get("output_table")

# Use the parameter as a table name
filtered_df.write.mode("overwrite").saveAsTable(output_table)
print(f"Wrote to {output_table}")

Pro tip: When passing parameters via dbutils.notebook.run, values are always strings. If you need a list or dict, serialize with json.dumps() on the sender side and json.loads() on the receiver.

Compare options / when to choose what

In a job, you can also define job parameters that are automatically available as widgets. This is the cleanest way to parameterize production runs.

Troubleshooting & edge cases

• Parameter values are always strings: If you pass 5 it comes as "5". You must convert to integer or date and handle ValueError for bad input.
• Widget name mismatch: If the parent passes region but the child expects region_name, the widget just returns the default (and you might not notice). Always log the received parameters.
• SQL injection: Never concatenate strings into SQL without validation. Use spark.table() or column references, or at least escape quotes.
• Widget persists across runs: If you don't remove widgets, old values might remain. Use dbutils.widgets.removeAll() at the end if needed.
• dbutils.notebook.run() timeout: The default is 120 seconds; set higher for long-running child notebooks, or you'll get a timeout error.

What you learned & what's next

You now understand what use parameters to parameterize notebooks means in Databricks: you can create reusable notebooks by exposing inputs through widgets, retrieving them with dbutils.widgets.get(), and passing them programmatically with dbutils.notebook.run(). You've completed a hands-on exercise that filters and writes data based on parameters, and you've seen how to avoid common pitfalls.

Next up: In the next lesson, you'll learn how to orchestrate multiple notebooks into a pipeline using Databricks workflows — where parameterization becomes essential for maintaining clean, flexible ETL jobs.

Keep your notebooks DRY (Don't Repeat Yourself), and parameterize early so you never maintain clones again.