How to Mock a User-Defined Function (UDF) in Python
Wrap a real UDF implementation with call logging to simulate and track invocations in a data pipeline.
Python code
31 linesfrom typing import Any, Callable
# Mock a user-defined function (UDF) that was previously complex or external
def mock_udf(name: str, implementation: Callable[..., Any], *, calls: list[Any]) -> Callable[..., Any]:
"""Wrap a real implementation with call logging to simulate a UDF."""
def wrapper(*args: Any, **kwargs: Any) -> Any:
calls.append((args, kwargs))
return implementation(*args, **kwargs)
return wrapper
# Real implementation of a simple UDF (e.g., a data transformation)
def real_udf(value: int) -> int:
return value * 2 + 1
if __name__ == "__main__":
# Track how many times the UDF is called
call_log: list[Any] = []
# Create the mocked version
mocked = mock_udf("transform_element", real_udf, calls=call_log)
# Simulate a data pipeline calling the UDF
values = [1, 2, 3, 4]
results = [mocked(v) for v in values]
print("Results:", results)
print("Number of calls:", len(call_log))
print("Call log:", call_log)
Output
Results: [3, 5, 7, 9]
Number of calls: 4
Call log: [((1,), {}), ((2,), {}), ((3,), {}), ((4,), {})]
How it works
The mock_udf function returns a wrapper that records each call's arguments in a shared list, then forwards the call to the real implementation. This simulates how a UDF engine might wrap user code to add observability. The call log lets you inspect invocation patterns without touching the UDF's logic. This pattern is useful when testing pipeline steps that depend on UDF call counts or inputs.
Common mistakes
- Forgetting to pass the call log list and accidentally using a new list per call.
- Not handling keyword arguments in the wrapper, leading to missed vectorized calls.
- Assuming the wrapper is thread-safe when multiple workers call the UDF concurrently.
Variations
- Use a decorator style with `functools.wraps` to preserve metadata.
- Return a tuple of (mocked_func, log) from a factory to encapsulate state.
Real-world use cases
- Testing PySpark UDFs by mocking them to count calls and verify input transformations.
- Instrumenting a legacy UDF during migration to log each invocation for auditing.
- Simulating expensive external UDFs in unit tests to avoid network or resource calls.
Sponsored
More from Big data & Spark
Keep learning
Related tutorials and quizzes for this topic.