How to Mock a Catalyst Logical Plan in Python
Build a small Python class that mimics Spark Catalyst's logical plan tree for teaching or testing query optimizations.
Python code
46 linesfrom typing import Any, Dict, List, Optional
class CatalystLogicalPlan:
"""A minimal mock of Catalyst's logical plan for teaching purposes."""
def __init__(self, node_type: str, **kwargs: Any) -> None:
self.node_type = node_type
self.attributes: Dict[str, Any] = kwargs
self.children: List["CatalystLogicalPlan"] = []
def add_child(self, child: "CatalystLogicalPlan") -> None:
self.children.append(child)
def output_columns(self) -> List[str]:
if "output" in self.attributes:
return self.attributes["output"]
if self.children:
return self.children[-1].output_columns()
return []
def explain(self, indent: int = 0) -> str:
prefix = " " * indent
details = ", ".join(f"{k}={v}" for k, v in self.attributes.items())
result = [f"{prefix}+- {self.node_type} ({details})"]
for child in self.children:
result.append(child.explain(indent + 1))
return "\n".join(result)
def build_mock_plan() -> CatalystLogicalPlan:
scan = CatalystLogicalPlan("Project", output=["id", "name", "salary"])
filter_node = CatalystLogicalPlan("Filter", predicate="salary > 50000")
aggregate = CatalystLogicalPlan("Aggregate", grouping_keys=["department"], agg_func="avg(salary)")
# Build tree: Aggregate <- Filter <- Scan
scan.add_child(CatalystLogicalPlan("Scan", table="employees"))
filter_node.add_child(scan)
aggregate.add_child(filter_node)
return aggregate
if __name__ == "__main__":
plan = build_mock_plan()
print(plan.explain())
print(f"Output columns: {plan.output_columns()}")
Output
+- Aggregate (grouping_keys=['department'], agg_func=avg(salary))
+- Filter (predicate=salary > 50000)
+- Project (output=['id', 'name', 'salary'])
+- Scan (table=employees)
Output columns: ['id', 'name', 'salary']
How it works
The CatalystLogicalPlan class stores a node type, arbitrary attributes, and child nodes to form a tree. explain recursively prints the tree with indentation, mimicking Spark's plan display. output_columns walks up from the leaf (Scan) to infer the final columns, since the Project node defines the output. This mock is pure Python with no Spark dependency, so you can test plan transformations or exercises without a cluster.
Common mistakes
- Forgetting to add the Scan node as a child of Project, causing incorrect output_columns
- Mutating attributes after build, breaking the plan's immutability (Catalyst plans are immutable)
- Not handling missing 'output' attribute gracefully, leading to empty column lists
Variations
- Use dataclasses with slots for lighter and more structured nodes
- Add a `copy` method to support immutable transformations like Catalyst's `transform` API
Real-world use cases
- Teaching Spark query optimization by visually inspecting logical plan transformations.
- Unit-testing custom optimizer rules without spinning up a full Spark session.
- Building a lightweight planning engine for a homebrew SQL-like DSL.
Sponsored
More from Big data & Spark
Keep learning
Related tutorials and quizzes for this topic.