How to Reduce Aggregate Counts from Mapped Chunks in Python

Combine a list of mapped chunk dictionaries into a single aggregated count dictionary using functools.reduce.

Easy Python 3.6+ Aug 9, 2026 Data pipelines & processing 14 views 0 copies

reduce aggregation dictionary mapreduce functools

Python code

22 lines

Python 3.6+

from functools import reduce
from collections import defaultdict

def aggregate_chunks(mapped_chunks):
    """Combine mapped chunk counts into a single aggregate dict."""
    return reduce(
        lambda acc, chunk: {
            **acc,
            **{k: acc.get(k, 0) + v for k, v in chunk.items()}
        },
        mapped_chunks,
        {}
    )

if __name__ == "__main__":
    chunks = [
        {"apple": 3, "banana": 2},
        {"apple": 1, "cherry": 4},
        {"banana": 5, "cherry": 1, "date": 2},
    ]
    result = aggregate_chunks(chunks)
    print(result)

Output

stdout

{'apple': 4, 'banana': 7, 'cherry': 5, 'date': 2}

How it works

The functools.reduce function applies a binary function cumulatively to the items in the list, starting with an empty initial accumulator. For each chunk, the lambda merges the accumulator with the chunk by summing values for common keys using acc.get(k, 0) + v. The dictionary unpacking {**acc, **{...}} creates a new merged dict for each step, which is fine for small datasets but can be replaced with an in-place update for performance. This pattern mirrors the reducer step in MapReduce and is useful for aggregating counts or sums from partitions of data.

Common mistakes

Using `acc[k] = acc.get(k, 0) + v` inside a loop instead of the pure lambda, still works but less functional
Forgetting the initial value `{}` in `reduce`, which makes the first chunk be used as the accumulator
Mutating the original chunks when using `defaultdict` in the reducer without copying

Variations

Using a `defaultdict(int)` and a for loop to accumulate in-place for better performance
Using `collections.Counter` and summing with `sum(chunks, Counter())` for a simpler aggregation

Real-world use cases

Merging word counts from different log files or map outputs in a batch processing pipeline.
Aggregating event counts from multiple partitions or shards before loading into a database.
Combining metrics or click counts from several service instances into a single total.

How to Reduce Aggregate Counts from Mapped Chunks in Python

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from Data pipelines & processing

Tutorials

Quizzes

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from Data pipelines & processing

Keep learning

Tutorials

Quizzes