Leverage Photon Engine

You've built your Delta Lake tables, tuned your cluster, and written clean DataFrame code — yet some queries just feel slow. The data is right there, the schema is perfect, but GROUP BY over millions of rows crawls. This is the classic moment when Databricks users discover the Photon engine: a native vectorized query engine that runs your SQL and DataFrame workloads at C++ speed without rewriting a single line of code. In this lesson, you'll learn how to leverage the Photon engine for query speed — when to enable it, how to verify it's working, and why it's one of the fastest wins in your Databricks performance toolbox.

The problem this lesson solves

Every distributed query engine has a bottleneck. In Apache Spark, the default execution engine compiles each query into Java bytecode, then evaluates rows one at a time using the whole-stage code generation technique. While this is far faster than naive interpreter loops, it still faces CPU and memory pressure: each row triggers object allocations, virtual method calls, and type checks. For CPU-bound workloads — aggregations, joins, filters, sorting — this overhead can account for 30–50% of total execution time.

The pain is real: your dashboard queries take 10 seconds when the data is only 2 GB. Your ELT jobs run for 40 minutes even though your cluster has 32 cores. You've tried repartitioning, bucketing, and caching — but the fundamental CPU bottleneck remains. That's exactly the problem Photon was built to solve.

Why now? Data volumes grow faster than CPU speeds. You can't just buy more cores to fix a CPU-bound pipeline indefinitely. Engine-level optimization is the only lever that scales without adding cost.

Core concept / mental model

Think of Spark's default engine as a generic delivery van: it can carry any package, but it stops at every house, checks the address manually, and handles each parcel one at a time. Photon, on the other hand, is a high-speed cargo train: it only runs on the standard-gauge railway (Delta Lake, Parquet, ORC), but it flies down the track carrying thousands of rows in a single freight car.

Photon is a native C++ vectorized execution engine integrated into Databricks Runtime (DBR). It replaces the Spark SQL and DataFrame execution for supported operations with vectorized processing: instead of processing one row at a time, it operates on batches of rows (often 1000+), using SIMD (Single Instruction, Multiple Data) CPU instructions to process multiple values in parallel.

The mental model breaks down into three parts:

1. Vectorization — Data is processed in columnar batches, not row-by-row. This reduces function call overhead and improves CPU cache utilization.
2. Native code — Written in C++, Photon avoids the JVM's garbage collection and object allocation pressure. Less GC pause = more predictable performance.
3. Compatibility — Photon is not a separate engine; it's a drop-in accelerator for existing Spark SQL and DataFrame APIs. Your code stays the same; only the execution changes.

Key insight: Photon is not a replacement for Spark — it's an accelerator inside Spark. Your existing notebooks, connectors, and SQL queries keep working; they just run faster.

How it works step by step

Photon doesn't require you to change your coding style. Instead, it works under the hood through three sequential steps once enabled:

1. Enable Photon on your cluster. In the Databricks UI, when creating or editing a cluster, select a Photon runtime version (they're labeled with "Includes Photon"). Alternatively, for existing clusters, toggle the Photon Acceleration checkbox in the cluster's advanced options. The cluster must be restarted for the change to take effect.

2. Write or run your existing query. Any SQL query, DataFrame operation, or spark.sql() call that uses supported operations (filters, joins, aggregations, sorting, window functions) will automatically be routed to Photon. You don't need to annotate your code or use a special API.

3. Verify via the Spark UI. When a query runs, the SQL tab shows whether the query executed with Photon. You'll see a small Photon badge or the physical plan will contain Photon nodes (e.g., PhotonSort, PhotonFilter). Also, the Query History in Databricks SQL shows performance metrics and a "Photon" flag for each query.

The execution path for a simple aggregation becomes:

• Your code (SQL or DataFrame) → Catalyst optimizer (logical plan) → Photon physical plan (vectorized operators) → C++ execution → result.

The Catalyst optimizer still handles cost-based optimizations (join reordering, predicate pushdown), but the physical execution — the heavy lifting — is done by Photon's native operators.

Efficiency tip: Photon is most effective when your queries are CPU-bound — that means the bottleneck is computation, not I/O. If your query is scanning terabytes from remote storage, Photon will still help, but the gain may be lower.

Hands-on walkthrough

Let's make this concrete. We'll create a modest dataset, run a query with Photon disabled, then enable Photon and measure the difference.

Step 1: Create a test dataset

Run the following in a Databricks notebook with a cluster that has Photon disabled (standard runtime).

from pyspark.sql import functions as F

# Create a DataFrame with 10 million rows
spark.range(1, 10000000).createOrReplaceTempView("numbers")

# Simulate a real analytical workload: group by modulo 1000 and compute aggregates
df = spark.sql("""
  SELECT id % 1000 AS group_id, 
         SUM(id) AS total, 
         AVG(id) AS avg_id, 
         COUNT(DISTINCT id) AS distinct_count
  FROM numbers
  GROUP BY 1
  ORDER BY 1
""")

# Cache in Delta for fair comparison
spark.sql("CREATE OR REPLACE TABLE sales_perf USING delta AS SELECT id, id % 1000 AS grp FROM numbers")

The CREATE TABLE writes the data to Delta, so later queries read from disk — a realistic scenario.

Step 2: Run and time the query (without Photon)

import time

start = time.time()
df = spark.sql("""
  SELECT grp, COUNT(*) as cnt, AVG(id) as avg_id
  FROM sales_perf
  WHERE id > 5000000
  GROUP BY grp
  ORDER BY grp
""")
result = df.collect()
print(f"Query runtime (no Photon): {time.time() - start:.3f} seconds")
print(result[:3])

Expected output (times will vary):

Query runtime (no Photon): 5.234 seconds
[(0, 4999, 7500498.0), (1, 4999, 7500999.0), (2, 4999, 7501499.0)]

Step 3: Enable Photon and rerun

1. Stop the cluster.
2. Edit the cluster: switch to a runtime version that says Includes Photon (e.g., DBR 12.2 LTS Photon).
3. Restart the cluster.
4. Re-run the exact same query from Step 2 (the Delta table persists).

import time

start = time.time()
df = spark.sql("""
  SELECT grp, COUNT(*) as cnt, AVG(id) as avg_id
  FROM sales_perf
  WHERE id > 5000000
  GROUP BY grp
  ORDER BY grp
""")
result = df.collect()
print(f"Query runtime (with Photon): {time.time() - start:.3f} seconds")
print(result[:3])

Expected output (again, times vary):

Query runtime (with Photon): 1.872 seconds
[(0, 4999, 7500498.0), (1, 4999, 7500999.0), (2, 4999, 7501499.0)]

In this example, Photon delivered a ~2.8x speedup. Your actual results depend on hardware, data size, and query complexity, but a 1.5x–3x improvement is typical for CPU-bound analytical queries.

Step 4: Verify Photon is active

After running any query, open the Spark UI → SQL tab. Find the query in the list. If Photon executed it, you'll see a green Photon label next to the query. You can also check the Physical Plan with:

df.explain()

If Photon is used, the physical plan will include operator names like PhotonSort, PhotonExchange, or PhotonHashedAggregate instead of the standard Sort or HashAggregate.

Compare options / when to choose what

Photon is a fantastic default for analytics, but it's not the only performance lever. Here's how it compares to other common tuning methods:

When to choose Photon:

• You're running interactive SQL dashboards or ad-hoc analytical queries on Delta tables.
• Your ETL pipelines use standard DataFrame/SQL transformations (filter, join, groupBy, window).
• You're on Databricks SQL (where Photon is automatically used for SQL endpoints) or on a compute cluster with Photon enabled.

When NOT to use Photon:

• Your workload is dominated by Python UDFs or complex Lambda expressions that run in Spark's JVM. Photon can't accelerate those parts (though it may speed the surrounding operations).
• You're doing streaming with stateful operations that aren't fully supported (check your runtime's Photon compatibility matrix).
• You need to use a non-Photon runtime for compliance reasons (usually rare).

Pro tip: Photon is included with most Databricks plans at no additional cost for SQL Warehouses. If you're paying for Databricks SQL, you're almost certainly already benefiting from Photon — check your query history to confirm.

Troubleshooting & edge cases

"My query doesn't use Photon"

Symptom: The SQL tab shows no Photon badge, or explain() has no Photon* operators.

Cause: Your cluster isn't running a Photon-enabled runtime. Check the cluster's runtime version string — it should contain "Photon" (e.g., 12.2 LTS Photon).

Fix: Restart the cluster with a Photon runtime. For SQL Warehouses, Photon is enabled by default; for notebooks, ensure the cluster has the checkbox Photon Acceleration selected (or use a Photon-labeled runtime).

"Photon doesn't speed up my query — sometimes it's slower"

Symptom: No improvement or worse performance after enabling.

Cause: Your query is not CPU-bound. Common causes: - The query is I/O bound (reading massive amounts of data from cloud storage without caching). - Your query uses a UDF which forces a fallback to JVM execution. The interoperation between Photon and UDFs can add overhead. - Your cluster is too small — Photon's vectorized operators need enough cores to shine.

Fix: Profile with the Spark UI. If the query spends most time in Scan or Storage, the bottleneck is I/O — consider Delta cache or data skipping. If you see a PythonUDF node, rewrite the logic using built-in Spark functions that Photon supports.

"Photon errors with 'Unsupported operation'"

Symptom: An exception or a warning that a specific operation is not supported by Photon.

Cause: Some esoteric SQL expressions or third-party connectors fall outside Photon's supported set (which is already very broad).

Fix: Photon gracefully falls back to standard Spark for unsupported operations — your query still runs. If performance is critical, try rewriting the operation using more basic built-ins (e.g., replace collect_list with array_agg if available, or use SQL instead of complex Python mashups).

"I see high memory usage after enabling Photon"

Symptom: Executor memory increases significantly.

Cause: Photon allocates memory for columnar batches. This is normal, but if you exceed memory, you may see spilling or OOM.

Fix: Increase spark.sql.adaptive.shuffle.maxNumPostShufflePartitions or add more executor memory. Also check that you don't have too many shuffle partitions (bump spark.sql.shuffle.partitions to a reasonable number like 200).

What you learned & what's next

You now understand how to leverage Photon engine for query speed. You've learned:

• The core problem: CPU overhead in row-based execution slows down analytics.
• The mental model: Photon = vectorized native execution engine inside Spark.
• How it works: Automatic acceleration for supported SQL/DataFrame operations, easy to enable via cluster settings.
• Hands-on: You measured a ~2.8x speedup on a synthetic query and verified Photon's presence in the Spark UI.
• When to use it: CPU-bound, standard analytical workloads — and when to avoid it (UDF-heavy or I/O-bound scenarios).

What's next? You're ready to dig into performance optimization patterns — for example, learning how to use Delta Lake Z-ORDER or Auto Optimize to reduce the amount of data Photon has to scan. These features compound with Photon: faster scan + faster compute = maximum query speed.

Final pro tip: Before enabling Photon for a critical pipeline, run a quick A/B test on a copy of your cluster. Enable Photon, run your top 10 production queries, and compare timings. In most cases, the decision will be an easy yes.

Now go enable Photon on your cluster and watch your queries fly — you've earned it.