Inspect Spark UI for Tuning

Your Spark job runs slower than a weekend batch job, and you have no idea why. You've guessed at spark.sql.shuffle.partitions, bumped the cluster size, and still the progress bar crawls. The Spark UI — that browser-based dashboard every Databricks notebook links to — holds the answer, but only if you know what to look for. In this lesson, you'll learn to inspect Spark UI for job tuning: read the stages, spot skew and spills, and make data-driven decisions that actually move the needle.

The problem this lesson solves

Spark applications are distributed systems, and distributed systems fail — or crawl — in ways that are impossible to debug with print() statements. You need a single pane of glass that shows you what every executor is doing, where data is shuffling, and where time is lost. That pane is the Spark UI, and it's built into every Databricks cluster.

The pain: you see a job that takes 20 minutes, but you don't know if the bottleneck is CPU, memory, disk I/O, or a single skewed partition. You tweak parameters blindly, hoping for a win. Without inspection, tuning is guesswork. This lesson turns guesswork into engineering.

By the end, you'll be able to open the Spark UI in Databricks, read the Jobs, Stages, and SQL tabs, identify the slowest stage, drill into task-level metrics, and apply targeted fixes like adjusting shuffle partitions or repartitioning skewed data.

Core concept / mental model

Think of the Spark UI as the flight recorder for your Spark application. Just as a pilot uses instruments to understand altitude, speed, and fuel, you use the Spark UI to understand:

• Jobs: a high-level action (like count() or write())
• Stages: a set of tasks that can run without shuffling data
• Tasks: the smallest unit of work, running on a single partition

💡 Pro tip: The Spark UI is not just for post-mortems. Use it during development to validate that your partitioning strategy is doing what you expect.

Here's the mental picture: your Spark job is an assembly line. Jobs are the overall product requests. Stages are the stations where work happens. Tasks are the workers at each station. The Spark UI shows you if one worker is overloaded (skew), if the line is idle waiting for parts (shuffle), or if the station is too slow (CPU or I/O bound).

The key insight: tuning is not about random parameter changes — it's about identifying the bottleneck stage and fixing that. The Spark UI tells you exactly which stage to attack.

How it works step by step

When you run a Spark job, the driver and executors send metrics to the Spark UI in near real-time. Here's how to navigate it in Databricks:

1. Open the Spark UI: In a notebook, click the Spark UI button next to the cluster dropdown — or go to the cluster page and click Spark UI. A new tab opens with the default Jobs page.
2. Read the Jobs page: You'll see a list of jobs with their status, duration, and progress bars. Click on a job to see its stages.
3. Analyze Stages: The Stages tab (within a job) shows a DAG (directed acyclic graph) of operations. Each stage shows metrics like Shuffle Read Size, Shuffle Write Size, Input Size, and Output Size. Look for the stage with the longest duration.
4. Drill into Tasks: Click on a stage to see the task table. Each row is a task with metrics: Duration, GC Time, Shuffle Read/Write, Spill (Memory/Disk). Look for variance — a skew if one task takes 10x longer.
5. Check Event Timeline: The Event Timeline tab shows executor activity over time. Long gray gaps indicate idle executors — a sign of poor parallelism.
6. Use the SQL tab: If your job uses DataFrames, the SQL tab shows the physical plan with metrics per operator. It links directly to stages, making it easy to map SQL to execution.

🔍 Focus on the stage with the highest duration. That's your bottleneck. Everything else is secondary.

Hands-on walkthrough

Let's make it concrete. We'll run a simple but realistic job on Databricks, then inspect the Spark UI to find and fix a skew issue.

Setup: create a skewed dataset

from pyspark.sql import functions as F

# Create a small but skewed dataset
skewed = spark.range(1_000_000)\
    .withColumn("key", F.when(F.col("id") < 900_000, F.lit("hot"))
        .otherwise(F.lit("cold")))\
    .repartition(100, "key")

# Simulate a join on the skewed key
dim = spark.range(10).withColumn("key", F.lit("hot"))

result = skewed.join(dim, "key")\
    .groupBy("key").count()

# Trigger the job - this will run all stages
result.collect()

This creates a classic skew: 90% of data lands on the hot key. Run it, then open the Spark UI.

What you'll see in the UI

• Jobs: One job for collect().
• Stages: Likely 3 stages: (1) range and repartition, (2) join and shuffle, (3) aggregate. The second stage will show a huge Shuffle Read Size.
• Tasks: In stage 2, you'll see a few tasks taking 10–20 seconds while others finish in 1 second. That's skew.

Apply a fix: salt the hot key

Instead of guessing, the UI data tells you to fix skew. Here's a salt-based repartition:

from pyspark.sql import functions as F

# Add a salt column based on the hot key
salted = skewed.withColumn("salt", F.rand() * 10)\
    .withColumn("salted_key", F.concat(F.col("key"), F.lit("_"), F.col("salt")))

# Repeat the dimension table to match
salt_dim = dim.crossJoin(F.range(10).withColumnRenamed("id", "salt"))\
    .withColumn("salted_key", F.concat(F.col("key"), F.lit("_"), F.col("salt")))

result_fixed = salted.join(salt_dim, "salted_key")\
    .groupBy("key").count()

result_fixed.collect()

Now, inspect the Spark UI again. The task durations in stage 2 should be roughly uniform — no single task dominates.

💡 Pro tip: Always check Spill (Memory/Disk) in the task metrics. If you see spills, your executors are running out of memory — a separate tuning knob.

Expected output

After the first run, the UI shows a max task duration of ~20s for one task. After the fix, the max task duration drops to ~2s. The overall job time should improve proportionally.

Compare options / when to choose what

Not every job benefits from aggressive tuning. Here's a decision guide:

When to tune vs. restructure: If the Spark UI shows a stage with a single task that's slow due to a filter, consider restructuring the data layout (e.g., Z-ordering on Delta Lake) rather than adding resources. The UI points you to the decision.

Troubleshooting & edge cases

Even with the Spark UI, things go wrong. Here are common pitfalls and fixes:

1. "I don't see the Spark UI button"

• Cause: You have a Serverless cluster or the notebook is not attached.
• Fix: Attach the notebook to an interactive cluster, or check your cluster's Advanced options — serverless may not expose the full UI.

2. "The UI shows a successful job but it's slow"

• Cause: The job is I/O bound, but the UI shows low CPU.
• Fix: Look at Input/Output metrics. If reading from cloud storage, consider caching or using Delta Lake with file pruning.

3. "Task durations are equal but job is slow"

• Cause: The shuffle phase dominates — look at Shuffle Read Time.
• Fix: Optimize join order, or increase shuffle partitions to reduce per-partition size (but not too many — overhead).

4. "Spark UI shows many jobs, but I only wrote once"

• Cause: Spark triggers multiple jobs for actions like collect() and write() when using multiple operations.
• Fix: Combine actions or cache() intermediate results to avoid recomputation.

5. "Spill metrics are high — what does that mean?"

• Cause: Executors ran out of memory and wrote to disk.
• Fix: Increase spark.executor.memory or reduce data per partition (increase partitions). Check the Event Timeline for garbage collection spikes.

⚠️ Common mistake: Don't set spark.sql.shuffle.partitions to 2000 blindly. The Spark UI will show increased overhead (task scheduling) that negates gains. Use the UI to see the effect, not just set and forget.

What you learned & what's next

You've now mastered the core skill of inspecting the Spark UI for job tuning. You can:

• Open and navigate the Spark UI in Databricks.
• Identify bottleneck stages using DAGs and task metrics.
• Detect data skew and apply salt-based fixes.
• Diagnose memory spills and adjust executor configuration.
• Compare tuning strategies and choose the right one based on evidence.

This forms the foundation for the next lesson: Optimizing shuffle partitions with AQE — where you'll use the Spark UI metrics you just learned to configure Adaptive Query Execution for automatic tuning.

Keep the Spark UI open during all your future jobs. It's your best friend in the Databricks journey.