Schedule Notebooks with Jobs

You've built a beautiful Databricks notebook that cleans data, computes aggregations, or trains a model—but if you still have to click Run manually every morning, you don't have a pipeline yet, you have a chore. Manually rerunning notebooks is error-prone, eats up engineering time, and means your dashboards and reports are only as fresh as your last click. In this lesson, you'll learn how to schedule notebooks with Databricks Jobs, turning your one-off code into reliable, automated workflows that run on time, every time—and you'll pick up the cluster management and failure-handling skills that make production ETL actually trustworthy.

The problem this lesson solves

Picture this: you've just deployed a notebook that transforms raw sales data into a clean Delta table. It works perfectly when you run it. But then the CEO wants the dashboard updated every morning at 7 AM, and your team needs the data fresh before standup. Suddenly your notebook is a problem, not a solution.

Manual runs don't scale

• Human error: You forget to run it on a holiday. The dashboard goes stale, and nobody notices until the afternoon.
• No history: If a run fails at 2 AM, you have no record of why, and no alert.
• No retries: A transient network blip crashes your run, and you don't know until the end of the day.
• Resource waste: You keep an interactive cluster running 24/7 just in case, burning Databricks units (DBUs) while it idles.

Databricks Jobs solve all of this. A job is a way to run one or more notebooks (or Python scripts, JARs, or SQL queries) on a schedule—or on demand—with built-in retries, notifications, and a full run history. You define what to run, when to run it, and what happens if it fails. From then on, the platform handles the orchestration, so you can focus on improving the logic instead of babysitting the runs.

Core concept / mental model

Think of a Databricks Job as a clock-driven postman: at the trigger time, it knocks on a cluster door, hands your notebook to Spark, and waits for the result. If the door is locked (cluster won't start) or the package is broken (your code throws), the postman tries again according to your rules, and if it still fails, it sends you a notification.

Key terms you'll meet

• Task: The unit of work—typically a notebook, but could also be a Python script, a SQL query, or a JAR.
• Cluster: the compute your job uses. You can use a job cluster (created fresh for each run, then terminated) or an existing all-purpose cluster (shared with interactive work).
• Schedule: A cron-style expression (e.g., 0 7 * * *) or a simple frequency (every 1 hour).
• Trigger: The event that starts a run—a time schedule, a file arrival, or a manual click.

Two types of compute for jobs

Pro tip: For production schedules, always use job clusters. They isolate your production code from interactive experiments and stop paying the moment the run finishes.

How it works step by step

Step 1: Prepare your notebook

Make sure your notebook is idempotent—running it twice gives the same result. For example, if you're writing to a Delta table, use saveAsTable with overwrite mode or create the table if not exists. A job that runs at 7 AM should not break on its second day.

Step 2: Create the job

In the Databricks UI, go to Workflows → Jobs → Create Job. You'll see a form where you give the job a name, choose a task type (Notebook), select your notebook, and configure the cluster.

Step 3: Configure the compute

• New job cluster: Pick a Databricks runtime version and a node type (e.g., Standard_DS3_v2). Set the minimum and maximum workers if you're using autoscaling—start small, like min=1, max=2, to control costs.
• Existing cluster: Choose from the dropdown, but remember the cost trade-off.

Step 4: Set the schedule

Under Schedule, you can choose a monthly, weekly, daily, or custom cron option. For example, a daily 7 AM run is 0 7 * * *. You can also set a timezone—important if your team is distributed.

Step 5: Configure notifications and retries

• Email notifications for start, success, or failure.
• Retries: Set a fixed number of retries (e.g., 2) and a delay (e.g., 5 minutes) before each retry.
• Timeout: Set a max run time (e.g., 60 minutes) to kill runaway jobs.

Step 6: Run and monitor

The job appears in Workflows → Jobs. You can trigger it manually ("Run now") or wait for the schedule. The Run history shows each attempt, its duration, and clickable output logs.

Hands-on walkthrough

Let's build a complete scheduled job from scratch.

Example 1: Create and run a simple scheduled job

First, create a notebook called daily_sales_report with the following code:

# daily_sales_report.py
from pyspark.sql import functions as F

# Read raw sales data
sales = spark.read.format("csv").option("header", "true").load("dbfs:/mnt/raw/sales.csv")

# Transform: parse date and compute daily revenue
results = (
    sales
    .withColumn("order_date", F.to_date("timestamp"))
    .groupBy("order_date", "region")
    .agg(F.sum("amount").alias("revenue"))
)

# Write results to a Delta table (overwrite to make idempotent)
results.write.mode("overwrite").format("delta").saveAsTable("default.daily_sales")

Now schedule it:

1. Go to Workflows → Jobs → Create Job.
2. Name it daily_sales_report_job.
3. Add a Notebook task, and select daily_sales_report.
4. Under Compute, choose New job cluster and accept the defaults (or pick a smaller node type for cost).
5. Under Schedule, set it to run every day at 7 AM (cron 0 7 * * *).
6. Click Create, then Run now to test it.

Expected output: In the Run history tab, you see a successful run. The Delta table daily_sales is created or updated.

Example 2: Add a retry for a flaky source

If your source is a network share that's sometimes unavailable, configure retries:

# The notebook is the same, but the job is configured in the UI (or via API).

Or use the Databricks CLI to create the job with retries:

databricks jobs create \
  --json '{
    "name": "daily_sales_report_job",
    "tasks": [
      {
        "task_key": "main",
        "notebook_task": {
          "notebook_path": "/workspace/Users/you@example.com/daily_sales_report"
        },
        "job_cluster_key": "default_cluster",
        "max_retries": 2,
        "min_retry_interval_millis": 300000
      }
    ],
    "job_clusters": [
      {
        "job_cluster_key": "default_cluster",
        "new_cluster": {
          "spark_version": "12.2.x-scala2.12",
          "node_type_id": "Standard_DS3_v2",
          "num_workers": 1
        }
      }
    ],
    "schedule": {
      "quartz_cron_expression": "0 7 * * *",
      "timezone_id": "UTC"
    }
  }'

Example 3: Parameterize your notebook

You can make your notebook accept parameters, so one job can serve multiple purposes:

# In the notebook, read parameters from the job: in a Databricks notebook, use dbutils.widgets
dbutils.widgets.text("source_path", "dbfs:/mnt/raw/sales.csv")
dbutils.widgets.text("target_table", "default.daily_sales")

source_path = dbutils.widgets.get("source_path")
target_table = dbutils.widgets.get("target_table")

sales = spark.read.csv(source_path, header=True)
# ... transform ...
sales.write.mode("overwrite").saveAsTable(target_table)

Then in the job task, set the parameters:

databricks jobs run-now --job-id 123 --notebook-params '{"source_path": "dbfs:/mnt/raw/sales_2025.csv", "target_table": "default.sales_2025"}'

Compare options / when to choose what

You have several ways to run notebooks on a schedule. Here's how they stack up:

When to use orchestrators: If your pipeline spans multiple systems (e.g., Databricks + a cloud SQL database), an orchestrator like Airflow might be better. But for jobs that live entirely in Databricks, native Jobs are simpler and more reliable.

Troubleshooting & edge cases

"Cluster failed to start"

• Check your quota limits on the cloud provider. Reduce the number of workers or choose a smaller node type.
• Invalid runtime version: Pick a supported Databricks runtime (e.g., 12.2 LTS).

"Run failed due to missing parameter"

• If you load a notebook that expects a widget, and the job doesn't pass it, the notebook throws an error. Always test the notebook manually first with the same parameters.

"My job runs but writes duplicate data"

• Not idempotent. If you append to a table instead of overwriting or upserting, every run adds a new batch. Use mode="overwrite" for full refreshes or Delta's merge for incremental.

"Run history is empty after schedule"

• Check the Schedule and timezone. If you use cron, make sure it's valid (e.g., 0 7 * * * not 0 7 * *).

"Job succeeded but notification didn't arrive"

• Verify email addresses and that notifications are enabled under Job settings. Some email providers block Databricks; check spam.

What you learned & what's next

You now understand the core idea behind scheduling notebooks with Databricks Jobs—from creating a job with a schedule, choosing the right compute, and configuring retries, to troubleshooting common pitfalls. You completed a hands-on exercise that automated a real-world ETL task, and you know how to parameterize notebooks for flexibility.

You're ready to move on to the next lesson in the Databricks track, where you'll dive deeper into orchestrating multi-task pipelines—joining multiple notebooks into a single job flow, handling dependencies, and building robust, production-grade data workflows.

Key takeaway: A scheduled job is more than a timer—it's a managed execution environment with retries, monitoring, and cost controls. Treat it as a production citizen.