How to Mock a UDAF Aggregate Function in Python
This code provides a minimal mock of a User-Defined Aggregate Function (UDAF), simulating the initialize-update-merge-finalize lifecycle with a defaultdict counter.
Python code
44 linesfrom collections import defaultdict
class MockUDAF:
"""A minimal mock of a User-Defined Aggregate Function.
Simulates aggregate lifecycle: initialize, update per row,
and finalize the result.
"""
def __init__(self):
self._buffer = defaultdict(int)
def initialize(self):
"""Reset internal state."""
self._buffer.clear()
def update(self, value):
"""Accumulate one input value."""
if value is not None:
self._buffer[value] += 1
def merge(self, other):
"""Merge another MockUDAF (simulates partial aggregates)."""
for key, count in other._buffer.items():
self._buffer[key] += count
def finalize(self):
"""Return the aggregate result."""
return dict(sorted(self._buffer.items(), key=lambda item: -item[1]))
@staticmethod
def run(values):
"""Convenience: full pipeline in one call."""
udaf = MockUDAF()
udaf.initialize()
for v in values:
udaf.update(v)
return udaf.finalize()
if __name__ == "__main__":
data = ["apple", "banana", "apple", "cherry", "banana", "apple", None]
result = MockUDAF.run(data)
print(result)
Output
{'apple': 3, 'banana': 2, 'cherry': 1}
How it works
The MockUDAF mimics Spark's UDAF lifecycle by providing initialize, update, merge, and finalize methods. It uses a defaultdict(int) to count occurrences of non-null values, and the merge method allows partial aggregates to be combined. The finalize method sorts items by count in descending order, making the result easy to read. The static run method chains the steps together for quick testing with small datasets.
Common mistakes
- Forgetting to import `defaultdict` from `collections`.
- Not handling `None` values in `update`, causing errors when counting.
- Mutating the buffer without clearing it in `initialize`.
Variations
- Use a regular dict with `get()` instead of `defaultdict` for Python < 3.9 compatibility.
- For a real UDAF, implement the same methods using Spark's `UserDefinedAggregateFunction` class.
Real-world use cases
- Unit testing aggregation logic before deploying to a Spark cluster.
- Prototyping a custom aggregation like weighted averages or percentile approximations locally.
- Teaching or demonstrating UDAF behavior without spinning up a distributed environment.
Sponsored
More from Big data & Spark
Keep learning
Related tutorials and quizzes for this topic.