How to Parse and Clean Text in Python

This code defines three helper functions to parse text into lowercase words, count unique word frequencies, and clean text by removing punctuation and extra whitespace.

Easy Python 3.9+ Aug 9, 2026 Strings & text 12 views 0 copies

text parsing string cleaning word frequency regex helpers

Python code

23 lines

Python 3.9+

def extract_words(text: str) -> list[str]:
    """Return a list of lowercase words from the given text."""
    return [word.lower() for word in text.split() if word.isalpha()]


def count_unique_words(text: str) -> dict[str, int]:
    """Return a dictionary with unique words and their frequencies."""
    words = extract_words(text)
    return {word: words.count(word) for word in set(words)}


def clean_text(text: str) -> str:
    """Remove extra whitespace and strip punctuation from the text."""
    import re
    cleaned = re.sub(r"[^\w\s]", "", text)
    return " ".join(cleaned.split())


if __name__ == "__main__":
    sample = "Hello, world! Hello everyone. This is a test, test text."
    print("Cleaned text:", clean_text(sample))
    print("Words:", extract_words(sample))
    print("Unique counts:", count_unique_words(sample))

Output

stdout

Cleaned text: Hello world Hello everyone This is a test test text
Words: ['hello', 'world', 'hello', 'everyone', 'this', 'is', 'a', 'test', 'test', 'text']
Unique counts: {'hello': 2, 'world': 1, 'everyone': 1, 'this': 1, 'is': 1, 'a': 1, 'test': 2, 'text': 1}

How it works

The extract_words function uses a list comprehension with split() to tokenize the text and isalpha() to keep only alphabetic words, then lowercases each word. The count_unique_words function builds a set of unique words and counts each occurrence with the count() method, returning a frequency dictionary. The clean_text function uses a regular expression to strip non-word/non-space characters and join() with split() to normalize whitespace.

Common mistakes

Using `isalpha()` on empty strings or words with digits, which returns False and drops valid numeric tokens
Forgetting to lowercase words before counting, leading to case-sensitive duplicates
Not importing `re` at module level, causing repeated imports inside functions

Variations

Use `collections.Counter` instead of a manual dictionary for more efficient frequency counting
Use `str.translate` with a translation table to remove punctuation without regex

Real-world use cases

Preprocessing user-generated comments before sentiment analysis or topic modeling.
Building a simple word-frequency report from log files or product reviews for quick insights.
Cleaning raw text scraped from web pages before storing it in a database.

How to Parse and Clean Text in Python

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from Strings & text

Tutorials

Quizzes

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from Strings & text

Keep learning

Tutorials

Quizzes