How to Implement MapReduce Word Count in Python Using a Dict

Simulate a MapReduce word count pipeline in Python with a mock dict, splitting text into words, shuffling, and reducing to frequency counts.

Easy Python 3.9+ Aug 9, 2026 Big data & Spark 16 views 0 copies

Python code

30 lines
Python 3.9+
def map_reduce_word_count(text: str) -> dict:
    """Simulate a MapReduce pipeline to count word frequencies."""
    # MAP phase: split into words and emit (word, 1) pairs
    mapped = []
    for word in text.lower().split():
        # Clean word of punctuation
        clean_word = ''.join(char for char in word if char.isalnum())
        if clean_word:
            mapped.append((clean_word, 1))
    
    # SHUFFLE phase: group by key
    shuffled = {}
    for key, value in mapped:
        if key not in shuffled:
            shuffled[key] = []
        shuffled[key].append(value)
    
    # REDUCE phase: sum values per key
    result = {}
    for key, values in shuffled.items():
        result[key] = sum(values)
    
    return result


if __name__ == "__main__":
    sample_text = "the quick brown fox jumps over the lazy dog the fox"
    counts = map_reduce_word_count(sample_text)
    print("Word count results:", counts)
    print("Total unique words:", len(counts))

Output

stdout
Word count results: {'the': 3, 'quick': 1, 'brown': 1, 'fox': 2, 'jumps': 1, 'over': 1, 'lazy': 1, 'dog': 1}
Total unique words: 8

How it works

This code mimics the MapReduce paradigm by breaking the process into three phases: map, shuffle, and reduce. The map phase splits the text into lowercase words and cleans them by removing punctuation, emitting (key, 1) pairs. The shuffle phase groups these pairs by key into a dictionary, collecting all values for each word. The reduce phase sums the values for each key to produce the final count. This pattern is conceptually similar to what Spark and Hadoop do at scale, but using simple Python dictionaries.

Common mistakes

  • Forgetting to lowercase text before splitting, causing case-sensitive counts
  • Not stripping punctuation, leading to words like 'fox.' being counted separately
  • Confusing the shuffle phase with sorting — it's just grouping here
  • Reusing a mutable list as a dictionary value without re-initializing

Variations

  1. Use collections.Counter after mapping to simplify the reduce phase
  2. Use a defaultdict(list) for the shuffle phase to avoid key checks

Real-world use cases

  • Prototyping a MapReduce job before deploying to a Spark cluster for log analysis.
  • Teaching the MapReduce concept in a classroom or training session without setting up distributed systems.
  • Performing quick, local text analytics on small datasets where a full big-data stack is overkill.

Sponsored

Run this sample

Open the browser IDE to tweak the example and see results without installing anything.

Open editor

More from Big data & Spark

Related tutorials and quizzes for this topic.