How to Mock a UDAF Aggregate Function in Python

This code provides a minimal mock of a User-Defined Aggregate Function (UDAF), simulating the initialize-update-merge-finalize lifecycle with a defaultdict counter.

Medium Python 3.9+ Aug 9, 2026 Big data & Spark 13 views 0 copies

Python code

44 lines
Python 3.9+
from collections import defaultdict

class MockUDAF:
    """A minimal mock of a User-Defined Aggregate Function.

    Simulates aggregate lifecycle: initialize, update per row,
    and finalize the result.
    """

    def __init__(self):
        self._buffer = defaultdict(int)

    def initialize(self):
        """Reset internal state."""
        self._buffer.clear()

    def update(self, value):
        """Accumulate one input value."""
        if value is not None:
            self._buffer[value] += 1

    def merge(self, other):
        """Merge another MockUDAF (simulates partial aggregates)."""
        for key, count in other._buffer.items():
            self._buffer[key] += count

    def finalize(self):
        """Return the aggregate result."""
        return dict(sorted(self._buffer.items(), key=lambda item: -item[1]))

    @staticmethod
    def run(values):
        """Convenience: full pipeline in one call."""
        udaf = MockUDAF()
        udaf.initialize()
        for v in values:
            udaf.update(v)
        return udaf.finalize()


if __name__ == "__main__":
    data = ["apple", "banana", "apple", "cherry", "banana", "apple", None]
    result = MockUDAF.run(data)
    print(result)

Output

stdout
{'apple': 3, 'banana': 2, 'cherry': 1}

How it works

The MockUDAF mimics Spark's UDAF lifecycle by providing initialize, update, merge, and finalize methods. It uses a defaultdict(int) to count occurrences of non-null values, and the merge method allows partial aggregates to be combined. The finalize method sorts items by count in descending order, making the result easy to read. The static run method chains the steps together for quick testing with small datasets.

Common mistakes

  • Forgetting to import `defaultdict` from `collections`.
  • Not handling `None` values in `update`, causing errors when counting.
  • Mutating the buffer without clearing it in `initialize`.

Variations

  1. Use a regular dict with `get()` instead of `defaultdict` for Python < 3.9 compatibility.
  2. For a real UDAF, implement the same methods using Spark's `UserDefinedAggregateFunction` class.

Real-world use cases

  • Unit testing aggregation logic before deploying to a Spark cluster.
  • Prototyping a custom aggregation like weighted averages or percentile approximations locally.
  • Teaching or demonstrating UDAF behavior without spinning up a distributed environment.

Sponsored

Run this sample

Open the browser IDE to tweak the example and see results without installing anything.

Open editor

More from Big data & Spark

Related tutorials and quizzes for this topic.