How to Mock a Catalyst Logical Plan in Python

Build a small Python class that mimics Spark Catalyst's logical plan tree for teaching or testing query optimizations.

Medium Python 3.9+ Aug 9, 2026 Big data & Spark 13 views 0 copies

Python code

46 lines
Python 3.9+
from typing import Any, Dict, List, Optional


class CatalystLogicalPlan:
    """A minimal mock of Catalyst's logical plan for teaching purposes."""
    
    def __init__(self, node_type: str, **kwargs: Any) -> None:
        self.node_type = node_type
        self.attributes: Dict[str, Any] = kwargs
        self.children: List["CatalystLogicalPlan"] = []

    def add_child(self, child: "CatalystLogicalPlan") -> None:
        self.children.append(child)

    def output_columns(self) -> List[str]:
        if "output" in self.attributes:
            return self.attributes["output"]
        if self.children:
            return self.children[-1].output_columns()
        return []

    def explain(self, indent: int = 0) -> str:
        prefix = "  " * indent
        details = ", ".join(f"{k}={v}" for k, v in self.attributes.items())
        result = [f"{prefix}+- {self.node_type} ({details})"]
        for child in self.children:
            result.append(child.explain(indent + 1))
        return "\n".join(result)


def build_mock_plan() -> CatalystLogicalPlan:
    scan = CatalystLogicalPlan("Project", output=["id", "name", "salary"])
    filter_node = CatalystLogicalPlan("Filter", predicate="salary > 50000")
    aggregate = CatalystLogicalPlan("Aggregate", grouping_keys=["department"], agg_func="avg(salary)")
    
    # Build tree: Aggregate <- Filter <- Scan
    scan.add_child(CatalystLogicalPlan("Scan", table="employees"))
    filter_node.add_child(scan)
    aggregate.add_child(filter_node)
    return aggregate


if __name__ == "__main__":
    plan = build_mock_plan()
    print(plan.explain())
    print(f"Output columns: {plan.output_columns()}")

Output

stdout
+- Aggregate (grouping_keys=['department'], agg_func=avg(salary))
  +- Filter (predicate=salary > 50000)
    +- Project (output=['id', 'name', 'salary'])
      +- Scan (table=employees)
Output columns: ['id', 'name', 'salary']

How it works

The CatalystLogicalPlan class stores a node type, arbitrary attributes, and child nodes to form a tree. explain recursively prints the tree with indentation, mimicking Spark's plan display. output_columns walks up from the leaf (Scan) to infer the final columns, since the Project node defines the output. This mock is pure Python with no Spark dependency, so you can test plan transformations or exercises without a cluster.

Common mistakes

  • Forgetting to add the Scan node as a child of Project, causing incorrect output_columns
  • Mutating attributes after build, breaking the plan's immutability (Catalyst plans are immutable)
  • Not handling missing 'output' attribute gracefully, leading to empty column lists

Variations

  1. Use dataclasses with slots for lighter and more structured nodes
  2. Add a `copy` method to support immutable transformations like Catalyst's `transform` API

Real-world use cases

  • Teaching Spark query optimization by visually inspecting logical plan transformations.
  • Unit-testing custom optimizer rules without spinning up a full Spark session.
  • Building a lightweight planning engine for a homebrew SQL-like DSL.

Sponsored

Run this sample

Open the browser IDE to tweak the example and see results without installing anything.

Open editor

More from Big data & Spark

Related tutorials and quizzes for this topic.