How to Simulate a MapReduce Mock with Combine Phase in Python
Simulates a MapReduce pipeline with a combiner that aggregates local counts per reducer to reduce network and compute overhead.
Python code
40 linesfrom collections import defaultdict
def map_phase(lines):
intermediate = defaultdict(list)
for line in lines:
for word in line.strip().lower().split():
intermediate[word].append(1)
return dict(intermediate)
def combine_phase(intermediate, num_reducers=3):
combined = defaultdict(list)
for word, counts in intermediate.items():
reducer_id = sum(ord(c) for c in word) % num_reducers
combined[reducer_id].append((word, sum(counts)))
return dict(combined)
def reduce_phase(combined):
results = {}
for reducer_id, word_counts in combined.items():
for word, total in word_counts:
results[word] = results.get(word, 0) + total
return dict(sorted(results.items()))
def map_reduce_mock(lines, num_reducers=3):
intermediate = map_phase(lines)
combined = combine_phase(intermediate, num_reducers)
return reduce_phase(combined), combined
if __name__ == "__main__":
lines = [
"hello world",
"hello python",
"world of python",
"python is powerful"
]
final_output, combiner_output = map_reduce_mock(lines)
print("Combiner local aggregates per reducer:")
for reducer_id, word_counts in sorted(combiner_output.items()):
print(f"Reducer {reducer_id}: {word_counts}")
print("Final reduced output:", final_output)Output
Combiner local aggregates per reducer:
Reducer 0: [('hello', 2), ('world', 2)]
Reducer 1: [('python', 3), ('is', 1)]
Reducer 2: [('of', 1), ('powerful', 1)]
Final reduced output: {'hello': 2, 'is': 1, 'of': 1, 'powerful': 1, 'python': 3, 'world': 2}How it works
The map phase tokenizes each line and emits key-value pairs with a count of 1. The combine phase groups these intermediate pairs by reducer ID (computed from the word's character sum modulo the number of reducers) and aggregates counts per word locally, simulating a combiner that reduces data shuffled to reducers. The reduce phase then sums the local aggregates across all reducers to produce the final global count. This mock mirrors the performance benefits of using combiners in real MapReduce frameworks like Hadoop.
Common mistakes
- Forgetting to call .lower() and .strip() on lines, causing inconsistent key casing or whitespace issues.
- Using a list instead of defaultdict(list) in the map phase, leading to KeyError on first append.
- Not sorting reducer ID keys when printing, which makes output non-deterministic in order.
- Assuming the combiner runs after the shuffle, but here it runs before, so design accordingly.
Variations
- Use a lambda-based hash for reducer assignment, e.g., hash(word) % num_reducers.
- Implement the combiner inside the reducer phase to avoid a separate step.
Real-world use cases
- Simulating MapReduce logic locally for unit testing before deploying to a cluster.
- Teaching distributed computing concepts with a lightweight, dependency-free example.
- Prototyping word count aggregations for log analysis before scaling to Spark.
Sponsored
More from Big data & Spark
Keep learning
Related tutorials and quizzes for this topic.