How to Use Broadcast Variables as Read-Only in PySpark (Mock Example)

Share a lookup dict across Spark executors with a broadcast variable and verify its read-only behavior in a local mock.

Easy Python 3.8+ Aug 9, 2026 Big data & Spark 13 views 0 copies

pyspark broadcast spark rdd read-only

Requires third-party packages — install first

pip install pyspark

Python code

27 lines

Python 3.8+

from pyspark import SparkContext, SparkConf

def main():
    conf = SparkConf().setAppName("BroadcastMock").setMaster("local[2]")
    sc = SparkContext(conf=conf)
    
    lookup = {"a": 1, "b": 2, "c": 3}
    broadcast_lookup = sc.broadcast(lookup)
    
    data = ["a", "b", "c", "a", "unknown"]
    rdd = sc.parallelize(data)
    
    result = rdd.map(lambda key: (key, broadcast_lookup.value.get(key, 0))).collect()
    
    print("Broadcast value (original):", broadcast_lookup.value)
    print("Lookup results:", result)
    
    # Show read-only behavior
    try:
        broadcast_lookup.value["d"] = 4
    except Exception as e:
        print("Modification attempt blocked:", type(e).__name__)
    
    sc.stop()

if __name__ == "__main__":
    main()

Output

stdout

Broadcast value (original): {'a': 1, 'b': 2, 'c': 3}
Lookup results: [('a', 1), ('b', 2), ('c', 3), ('a', 1), ('unknown', 0)]
Modification attempt blocked: TypeError

How it works

The sc.broadcast(lookup) call ships the dict once to each executor instead of serializing it with every task. Inside the map lambda, broadcast_lookup.value returns the shared dict, and .get(key, 0) safely falls back to 0 for missing keys. The read-only guarantee comes from PySpark wrapping the value so direct mutation raises a TypeError; attempts to modify blocked with an exception. After collect(), results are gathered to the driver, letting us print the lookup pairs. Always call sc.stop() to release the SparkContext and executor resources.

Common mistakes

Forgetting `broadcast_lookup.value` inside executors — the object itself is not serializable.
Trying to mutate the broadcast value — it raises an error by design; keep lookups immutable.
Calling `collect()` on huge RDDs — collect everything to the driver only for small mock data.
Creating a new SparkContext per function call without stopping it, leaking resources.

Variations

Use `sc.broadcast(lookup, sc=sc)` in older PySpark versions for explicit context.
Replace the manual map with `data.map(lambda k: (k, lookup.get(k, 0)))` if the dict fits the driver — but broadcast wins for large lookups.

Real-world use cases

Distributing a reference table (e.g., country-to-timezone mapping) across Spark workers for join-free enrichment.
Caching a trained ML model's label dictionary, so every partition during prediction reads the same mapping.
Broadcasting configuration rules (e.g., rate limits per API key) to avoid shipping them with each streaming micro-batch.

How to Use Broadcast Variables as Read-Only in PySpark (Mock Example)

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from Big data & Spark

Tutorials

Quizzes

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from Big data & Spark

Keep learning

Tutorials

Quizzes