Orchestrate Multi-Task Jobs

You've built notebooks that transform data beautifully, but in production, a single dataset rarely flows through one script. Real pipelines are chains: ingest, clean, join, aggregate, publish — and each step depends on the one before it. If you trigger these steps manually or glue them together with brittle shell scripts, you'll spend more time babysitting failures than building features. In this lesson, you'll learn how to orchestrate multi-task jobs with dependencies in Databricks — turning a pile of notebooks into a single, reliable, resumable workflow that runs itself.

The problem this lesson solves

Manually running notebooks in sequence might work for a demo, but it falls apart in production. Imagine you have three notebooks: 01_ingest, 02_clean, and 03_aggregate. Running them by hand means you're the orchestrator — you have to remember the order, wait for each to finish, and check for errors. Miss a step or run them in the wrong order, and your data is garbage. That's the orchestration problem: coordinating multiple tasks with dependencies and ensuring they run in the right sequence with the right resources.

Here's what makes this a real pain point for data engineers:

• Task ordering: Notebook B depends on output from A. If A fails, B must not run.
• Resource management: Each task might have different cluster requirements (e.g., a heavy join needs more workers).
• Observability: When something fails, you need to know which task and why — not just 'the pipeline is down'.
• Retry and recovery: You don't want to re-run the entire pipeline when a single task fails; you want to restart from the failing point.

Without proper orchestration, you end up with ad-hoc scripts, cron jobs, and a lot of manual intervention. Databricks Jobs solves this by letting you define a multi-task job with explicit dependencies, so the platform handles sequencing, retries, and monitoring.

Core concept / mental model

Think of a Databricks Job as a pipeline of tasks — a directed acyclic graph (DAG). Each task is a unit of work (notebook, JAR, SQL query, or Python script). Dependencies define the edges: Task B can only start after Task A succeeds. This is exactly how a build system like Make works, but designed for data workloads.

Key terms

• Task: A single unit of work in a job. Could be a notebook, a Spark submit, a SQL query, or a Python wheel.
• Dependency: A directed relationship between tasks. A task can depend on one or more other tasks.
• Job: A container for tasks and their dependencies, plus scheduling and notification settings.
• Task values: Outputs that a task can emit, which downstream tasks can consume as parameters — like a handoff.

To make this concrete: imagine your data pipeline as an assembly line. Each station (task) does one thing (clean, join, aggregate). You can't put the car on the body shop until the frame is welded. The assembly line is your job; the dependencies are the conveyor belts that only move when the previous station is done.

Why DAGs matter

DAGs allow parallelism and isolation. Tasks with no dependencies can run concurrently, speeding up your pipeline. If one task fails, only its downstream tasks are blocked — you can fix and retry just that branch. This is a huge improvement over a linear script where one failure kills everything.

How it works step by step

Creating a multi-task job in Databricks involves four logical steps:

1. Define tasks: Each task specifies a notebook or other workload, its parameters, and which cluster to use (or a serverless compute).
2. Set dependencies: For each task, declare which tasks must succeed before it starts. This creates the DAG.
3. Configure job-level settings: Schedule, timeout, retry policy, email notifications.
4. Run and observe: Trigger the job, monitor task status, and use the UI to restart failed tasks.

Task types

Databricks Jobs support several task types:

• Notebook: The most common — runs a .py or .ipynb notebook.
• SQL query / DBT: Run a SQL query or dbt transformation.
• Python wheel / JAR: Execute a packaged application.
• Delta Live Tables: A declarative pipeline.
• Run a job: Trigger another job (parent-child pattern).

How dependencies behave

The platform runs tasks in topological order. If Task A has two children (B and C), both B and C start as soon as A succeeds. If A fails, B and C are marked skipped. You can also add retry policies per task or at the job level.

Task value passing

A crucial feature is task values: a task can write key-value pairs (max 64 per task, 1KB each) that downstream tasks can read and inject as notebook parameters. This allows a clean task to hand a file path or a metric to the next task, making your pipelines truly dynamic.

Hands-on walkthrough

Let's build a real multi-task job. You'll create two notebooks and wire them together with dependencies.

Prerequisites

• A Databricks workspace (Community Edition works)
• Basic knowledge of notebooks and SQL

Step 1: Create the source notebook

Create a notebook named ingest_and_clean and paste this code:

# Databricks notebook source
# Ingest and clean raw data into a Delta table

# Simulate reading from a source
raw_df = spark.range(100).selectExpr("id", "cast(id % 5 as int) as partition")

# Write to a temp Delta location
clean_df = raw_df.filter("id > 10")  # drop some rows
clean_df.write.format("delta").mode("overwrite").saveAsTable("clean_data")

# Emit a task value describing what we did
import json
dbutils.jobs.taskValues.set(key="row_count", value=clean_df.count())
print(f"Cleaned {clean_df.count()} rows")

Step 2: Create the downstream notebook

Create aggregate_and_report that reads the table and creates a summary. It also reads a task value from the previous task:

# Databricks notebook source
# Consume the clean table and produce an aggregate

# Read the table produced by the previous task
clean_df = spark.table("clean_data")

# Aggregate
summary_df = clean_df.groupBy("partition").count()
summary_df.show()

# Read the task value set by the upstream task
from pyspark.sql import Row
row_count = dbutils.jobs.taskValues.get(taskKey="row_count")
print(f"Upstream row count: {row_count}")

# Write final report
summary_df.write.format("delta").mode("overwrite").saveAsTable("partition_summary")

Step 3: Create the job

In the Databricks UI:

1. Go to Workflows → Jobs → Create Job.
2. Name it multi_task_demo.
3. Add a task ingest_clean of type Notebook, pointing to ingest_and_clean. Choose a cluster (or Auto).
4. Add a second task aggregate_report of type Notebook, pointing to aggregate_and_report.
5. In the Depends on dropdown for aggregate_report, select ingest_clean. This creates the edge.
6. Set a schedule (or leave manual) and optionally configure email notifications.
7. Click Run now.

Expected output

After the run completes, you should see in the run details:

• Task ingest_clean succeeded.
• Task aggregate_report succeeded and ran after the first.
• In the aggregate_report notebook log:

+---------+-----+
|partition|count|
+---------+-----+
|        0|   18|
|        1|   18|
|        2|   18|
|        3|   18|
|        4|   18|
+---------+-----+

Upstream row count: 90

If you kill ingest_clean or make it fail, aggregate_report will be skipped, and the run will show a failure status.

Try a parallel branch

Add a third notebook quality_checks and create a task with no dependency on ingest_clean — it runs in parallel. Notice how the UI shows a DAG with two branches diverging from the start.

Compare options / when to choose what

Databricks offers multiple ways to run multi-step workloads. You already know how to orchestrate multi-task jobs with dependencies; here's how it compares to alternatives.

When to choose what: Use multi-task jobs when your pipeline is purely Databricks-native and you want simplicity plus reliability. Choose DLT if your workload is mostly streaming or you value automatic dependency management. Use an external orchestrator when you need to orchestrate across multiple cloud services or want a single control plane for everything.

Troubleshooting & edge cases

Even with good design, things go wrong. Here are common pitfalls and how to fix them.

1. Task fails but downstream runs

This should never happen if dependencies are set. If it does, check that you didn't disable the depends on relation accidentally — e.g., you set the dependency in the wrong direction. Verify the DAG in the job UI.

2. Task values not readable

Task values are only available to tasks that depend on the source task. If aggregate_report tries to read a value from ingest_clean but the dependency is missing, you'll get an error. Ensure the dependency exists and that the task actually ran successfully.

3. Notebook fails intermittently

Network or cluster issues can cause random failures. Configure retries (e.g., 2 retries with a 5-minute timeout) at the job level or per task. The platform will retry the failing task automatically, not the whole job.

4. Cluster cold starts slow down the job

Each task can use a different cluster, and spinning up a new cluster takes time. Use job-level cluster or serverless compute to avoid repeated spin-ups. Alternatively, set tasks to share a cluster if they're compatible.

5. Data skew breaks the aggregation

The partition column in our example is evenly distributed. In the real world, skewed keys cause stragglers. Use salt keys or repartition before aggregation to balance load.

6. Dependency cycle

If you accidentally create a cycle (A depends on B, B depends on A), the job will fail validation. The UI shows a warning — remove one edge.

What you learned & what's next

You can now orchestrate multi-task jobs with dependencies in Databricks. You understand the DAG mental model, how to set task dependencies in the UI, how to pass task values, and how to compare this approach to alternatives like DLT or Airflow. You also know common troubleshooting steps, from fixing dependencies to handling intermittent failures.

Next in the track: You'll build on this foundation by learning about scheduling and monitoring — setting up recurring runs, alerting on failures, and using the Jobs API to manage your pipelines programmatically. That's the key to making your orchestrated jobs reliable and self-sufficient.

For now, reinforce what you've learned by trying the practice exercise below.