Monitor Databricks Jobs

You've built and scheduled a Databricks job. It runs, it fails, it succeeds — but if you can't see why it behaved the way it did, you're flying blind. Without a clear view of run history, every failure is a mystery, every slow stage is a guess, and every retry is a gamble. This lesson shows you how to monitor jobs and view run history so you can track performance, debug failures with confidence, and keep your workflows healthy.

The problem this lesson solves

When a scheduled Databricks job fails at 2 AM, you need answers fast: Did it fail due to a code error, a cluster issue, or a data quality problem? How long did the last successful run take? Was that performance trend normal or a sign of trouble? If you rely on manual checks and scattered logs, you'll waste hours and risk downstream outages.

Monitoring jobs and viewing run history is your window into every execution: start and end times, status, cluster details, Spark UI, and logs. This lesson gives you the skills to observe your jobs systematically, so you can spot issues before they escalate and optimize confidently.

Core concept / mental model

Think of Databricks jobs as recipes and runs as each time you cook that recipe. The run history is your kitchen notebook — every attempt, success or failure, with timestamps, ingredients (cluster config), and notes (logs). Just as a chef reviews past attempts to perfect a dish, you review run history to perfect your data pipelines.

Key terms:

• Job: A definition of work — tasks, schedule, cluster config, and notifications.
• Run: A single execution of a job. A job can have many runs over time.
• Run history: The chronological list of all runs for a job, with statuses and metadata.
• Run ID: A unique identifier for each run, crucial for debugging and referencing in logs.

Pro tip: Always note the run ID when debugging. It's your main handle for correlating logs, metrics, and API calls.

How it works step by step

The Databricks UI and API track every run automatically. Here's the flow:

1. Trigger: A run starts due to a schedule, manual trigger, or API call.
2. Execution: The job's tasks run on a cluster. The platform records status changes (Pending, Running, Succeeded, Failed, Canceled).
3. Logging: Each run's stdout/stderr and system logs are stored and linked to the run.
4. Observation: You access run history via the UI (Jobs tab → job → Runs) or via API (runs/list).
5. Drill-down: For a specific run, you can open the Spark UI, view metrics, and inspect logs to diagnose.

This loop — trigger, execute, log, observe — is continuous, and monitoring is your active role in it. You're not just watching; you're gathering intelligence for decisions: Do I need to scale the cluster? Is the job schema drift-prone? Should I set a retry policy?

Hands-on walkthrough

Monitor jobs in the Databricks UI

Navigate to the Jobs tab, click your job name, and you'll see a Runs list. Each row shows status, start/end times, duration, and a link to the run details.

Click a run to see:

• Run status: Succeeded, Failed, Canceled, or Pending.
• Timing: Start, end, and duration.
• Cluster: Which cluster or cluster policy was used.
• Spark UI: For in-depth stage and task metrics.
• Logs: Stdout, stderr, and system logs.

Use the Databricks CLI to list runs

The CLI gives you scriptable access. For example:

# Set your Databricks host and token
export DATABRICKS_HOST=https://your-workspace.cloud.databricks.com
export DATABRICKS_TOKEN=your-personal-access-token

# List the runs for a specific job (replace JOB_ID)
databricks runs list --job-id 1234 --limit 10

Example output:

Run ID   Status      Start Time               Duration
5678     SUCCESS     2024-05-01 10:00:00     15m 30s
5679     FAILED      2024-05-01 11:00:00     2m 10s
5680     PENDING     2024-05-01 12:00:00     -

Programmatic monitoring with the API

For automated monitoring, use the Databricks REST API from Python:

import requests
import os

host = os.environ["DATABRICKS_HOST"]
token = os.environ["DATABRICKS_TOKEN"]
job_id = 1234

response = requests.get(
    f"{host}/api/2.1/jobs/runs/list",
    headers={"Authorization": f"Bearer {token}"},
    params={"job_id": job_id, "limit": 5},
)
response.raise_for_status()

for run in response.json().get("runs", []):
    run_id = run["run_id"]
    status = run["state"]["life_cycle_state"]
    result = run["state"]["result_state"] if "result_state" in run["state"] else "--"
    print(f"Run ID {run_id}: {status}, result {result}")

Output (example):

Run ID 5678: TERMINATED, result SUCCESS
Run ID 5679: TERMINATED, result FAILED
Run ID 5680: PENDING, result --

This lets you alert on failures, track SLA times, and build dashboards.

Compare options / when to choose what

Here's a comparison of the three primary ways to monitor jobs and view run history:

When to choose what: Use the UI for quick checks and root-cause analysis. Use the CLI for simple one-off queries or cron jobs. Choose the API when you need custom dashboards, alerting, or integration with existing systems.

Troubleshooting & edge cases

• Job shows 'Skipped': This usually means the run was triggered but an upstream task failed. Check the parent run's status and logs.
• Run stuck in 'Pending': The cluster may be starting or resources are unavailable. Check cluster status and retry the run.
• Logs not visible: Ensure you have permission to view job logs. Also check if the cluster was terminated; logs may be lost.
• API returns empty list: Confirm you're using the correct API version (e.g., 2.1) and that the job ID is valid. Check your token has jobs:read scope.
• Timeouts in long-running runs: Set timeouts at the task level to avoid infinite runs. Use the 'Timeout' field in job settings.

Pro tip: Set up email or Slack notifications in the job properties so you get immediate alerts on failure. You can monitor proactively, not reactively.

What you learned & what's next

You can now confidently monitor jobs and view run history in Databricks. You understand the UI, CLI, and API approaches, how to drill into run details, and how to troubleshoot common issues. You've learned to check status, timing, logs, and Spark UI — all essential for maintaining reliable data pipelines.

Next lesson: In the upcoming module, you'll learn how to automate job retries and alerts using job properties and notification systems. This builds on your monitoring skills to create self-healing workflows.

Keep your run history knowledge sharp — it'll serve you well in every future Databricks project.