How to Mock Partition Pruning in Python

A dataclass-based mock that filters partitions by year and month to emulate Spark's partition pruning logic.

Easy Python 3.9+ Aug 9, 2026 Big data & Spark 15 views 0 copies

spark partition dataclass filtering mock big-data

Python code

34 lines

Python 3.9+

from dataclasses import dataclass
from typing import List


@dataclass(frozen=True)
class Partition:
    id: int
    year: int
    month: int


class PartitionPruner:
    """Mock partition pruning: only keep partitions that match the filter."""
    def __init__(self, partitions: List[Partition]):
        self._partitions = partitions

    def prune(self, year: int, month: int) -> List[Partition]:
        return [p for p in self._partitions if p.year == year and p.month == month]


if __name__ == "__main__":
    candidates = [
        Partition(1, 2023, 1),
        Partition(2, 2023, 2),
        Partition(3, 2023, 3),
        Partition(4, 2024, 1),
        Partition(5, 2024, 2),
    ]

    pruner = PartitionPruner(candidates)
    selected = pruner.prune(year=2023, month=2)

    print("Selected partition IDs:", [p.id for p in selected])
    print("Selected partition details:", selected)

Output

stdout

Selected partition IDs: [2]
Selected partition details: [Partition(id=2, year=2023, month=2)]

How it works

PartitionPruner.prune uses a list comprehension to keep only partitions whose year and month match the filter arguments. The frozen=True dataclass makes Partition hashable and immutable, which is safe for representing metadata objects. This pattern mirrors how Spark's partition pruning eliminates files or directories that don't satisfy a predicate before reading data. It's a lightweight, dependency-free way to test filtering logic locally without a cluster.

Common mistakes

Using `any()` or `all()` instead of an exact `==` match on both fields
Forgetting to freeze the dataclass so partitions can be used in sets or as keys
Filtering on one field only when the filter requires both year and month
Not returning a new list, mutating the original instead

Variations

Use `filter(lambda p: p.year == year and p.month == month, self._partitions)` for a generator-style filter
Implement pruning with a dict keyed by `(year, month)` for O(1) lookups

Real-world use cases

Unit-testing Spark partition pruning logic in a data pipeline before deploying to a cluster.
Simulating Hive-style partition filters in a mock catalog for query planning tools.
Verifying time-range filters on partitioned event logs in a local ETL prototype.

How to Mock Partition Pruning in Python

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from Big data & Spark

Tutorials

Quizzes

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from Big data & Spark

Keep learning

Tutorials

Quizzes