How to Process Text in Python: Normalize Whitespace and Count Words

A beginner-friendly function that normalizes whitespace in a string and counts total and unique words using Python's standard library.

Easy Python 3.9+ Aug 9, 2026 Strings & text 13 views 0 copies

strings text-processing word-count beginners

Python code

27 lines

Python 3.9+

def process_text(text):
    """Basic text processing: normalize whitespace and count words."""
    normalized = " ".join(text.split())
    word_count = len(normalized.split())
    char_count = len(normalized)
    
    # Count unique words
    unique_words = set(normalized.lower().split())
    unique_count = len(unique_words)
    
    return {
        "original": text,
        "normalized": normalized,
        "word_count": word_count,
        "char_count": char_count,
        "unique_word_count": unique_count
    }


if __name__ == "__main__":
    sample = "  Hello   world!   This   is a   simple   text.   "
    result = process_text(sample)
    print(f"Original: '{result['original']}'")
    print(f"Normalized: '{result['normalized']}'")
    print(f"Words: {result['word_count']}")
    print(f"Characters: {result['char_count']}")
    print(f"Unique words: {result['unique_word_count']}")

Output

stdout

Original: '  Hello   world!   This   is a   simple   text.   '
Normalized: 'Hello world! This is a simple text.'
Words: 7
Characters: 34
Unique words: 7

How it works

The text.split() call splits on any whitespace (spaces, tabs, newlines) and discards empty tokens, which removes leading, trailing, and repeated spaces. Joining the tokens back with ' '.join(...) produces a clean, single-spaced string. Counting words is just len(normalized.split()), and converting to lowercase before using set() makes the unique-word count case-insensitive. The function returns a dictionary so all derived values stay paired with the original input.

Common mistakes

Using `text.replace(' ', '')` instead of `split()` and `join()` to remove extra spaces
Not lowercasing before counting unique words, so 'Hello' and 'hello' count as different
Confusing character count with word count when counting text length

Variations

Use `re.sub('\s+', ' ', text).strip()` for regex-based whitespace normalization
Use `collections.Counter(normalized.lower().split())` to also get per-word frequencies

Real-world use cases

Cleaning user-submitted comments before storing them in a database.
Pre-processing raw text for NLP tasks like tokenization or classification.
Building word-frequency reports from log files or social media posts.

How to Process Text in Python: Normalize Whitespace and Count Words

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from Strings & text

Tutorials

Quizzes

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from Strings & text

Keep learning

Tutorials

Quizzes