Scatter Gather Aggregate Pattern in Python

Simulates a scatter/gather/aggregate pattern by distributing work across items, gathering results, and aggregating them.

Easy Python 3.9+ Aug 9, 2026 Microservices patterns 13 views 0 copies

scatter-gather aggregation pattern distributed

Python code

19 lines

Python 3.9+

import random

def process_items(items, scatter_fn, gather_fn, aggregate_fn):
    """Simple scatter/gather/aggregate pattern simulation."""
    scattered = [scatter_fn(item) for item in items]
    gathered = [gather_fn(item) for item in scattered]
    return aggregate_fn(gathered)

if __name__ == "__main__":
    data = list(range(1, 11))
    random.seed(42)
    
    result = process_items(
        data,
        scatter_fn=lambda x: x * 2,
        gather_fn=lambda x: x + random.randint(0, 5),
        aggregate_fn=lambda values: sum(values) / len(values)
    )
    print(f"Average of processed values: {result:.2f}")

Output

stdout

Average of processed values: 52.35

How it works

The scatter phase transforms each input item independently, simulating parallel or distributed processing. The gather phase collects results from the scattered items, often representing network or I/O boundaries where randomness can introduce variability. The aggregate phase combines all gathered results into a single final value, mimicking a reduce step. Using Python's higher-order functions allows easy swapping of logic in each phase for testing or changing requirements.

Common mistakes

Assuming scatter/gather phases are thread-safe when using shared state
Forgetting to handle failures in individual scatter or gather calls
Mixing up the order phases: gather before scatter breaks the pattern

Variations

Use concurrent.futures.ThreadPoolExecutor to run scatter/gather in parallel
Implement scatter as a map operation and gather as a reduce using functools.reduce

Real-world use cases

Fanning out database queries to shards and aggregating results for a report.
Sending requests to multiple microservices and combining their responses.
Running parallel image processing tasks and computing summary statistics.

Scatter Gather Aggregate Pattern in Python

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from Microservices patterns

Tutorials

Quizzes

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from Microservices patterns

Keep learning

Tutorials

Quizzes