How to Mock a User-Defined Function (UDF) in Python

Wrap a real UDF implementation with call logging to simulate and track invocations in a data pipeline.

Easy Python 3.9+ Aug 9, 2026 Big data & Spark 13 views 0 copies

Python code

31 lines
Python 3.9+
from typing import Any, Callable


# Mock a user-defined function (UDF) that was previously complex or external
def mock_udf(name: str, implementation: Callable[..., Any], *, calls: list[Any]) -> Callable[..., Any]:
    """Wrap a real implementation with call logging to simulate a UDF."""
    def wrapper(*args: Any, **kwargs: Any) -> Any:
        calls.append((args, kwargs))
        return implementation(*args, **kwargs)
    return wrapper


# Real implementation of a simple UDF (e.g., a data transformation)
def real_udf(value: int) -> int:
    return value * 2 + 1


if __name__ == "__main__":
    # Track how many times the UDF is called
    call_log: list[Any] = []

    # Create the mocked version
    mocked = mock_udf("transform_element", real_udf, calls=call_log)

    # Simulate a data pipeline calling the UDF
    values = [1, 2, 3, 4]
    results = [mocked(v) for v in values]

    print("Results:", results)
    print("Number of calls:", len(call_log))
    print("Call log:", call_log)

Output

stdout
Results: [3, 5, 7, 9]
Number of calls: 4
Call log: [((1,), {}), ((2,), {}), ((3,), {}), ((4,), {})]

How it works

The mock_udf function returns a wrapper that records each call's arguments in a shared list, then forwards the call to the real implementation. This simulates how a UDF engine might wrap user code to add observability. The call log lets you inspect invocation patterns without touching the UDF's logic. This pattern is useful when testing pipeline steps that depend on UDF call counts or inputs.

Common mistakes

  • Forgetting to pass the call log list and accidentally using a new list per call.
  • Not handling keyword arguments in the wrapper, leading to missed vectorized calls.
  • Assuming the wrapper is thread-safe when multiple workers call the UDF concurrently.

Variations

  1. Use a decorator style with `functools.wraps` to preserve metadata.
  2. Return a tuple of (mocked_func, log) from a factory to encapsulate state.

Real-world use cases

  • Testing PySpark UDFs by mocking them to count calls and verify input transformations.
  • Instrumenting a legacy UDF during migration to log each invocation for auditing.
  • Simulating expensive external UDFs in unit tests to avoid network or resource calls.

Sponsored

Run this sample

Open the browser IDE to tweak the example and see results without installing anything.

Open editor

More from Big data & Spark

Related tutorials and quizzes for this topic.