Text Processor with Dictionaries and Sets in Python
Build a simple text processor that counts word frequencies with a dictionary and tracks unique words with a set.
Python code
29 linesdef analyze_text(text):
words = text.lower().split()
word_freq = {}
unique_words = set()
for word in words:
clean_word = word.strip('.,!?;:')
if clean_word:
word_freq[clean_word] = word_freq.get(clean_word, 0) + 1
unique_words.add(clean_word)
return word_freq, unique_words
def get_top_words(word_freq, n=3):
sorted_words = sorted(word_freq.items(), key=lambda x: x[1], reverse=True)
return sorted_words[:n]
if __name__ == "__main__":
sample_text = "The quick brown fox jumps over the lazy dog. The dog sleeps!"
word_freq, unique_words = analyze_text(sample_text)
print("Word frequency dictionary:")
print(word_freq)
print("\nUnique words set:")
print(unique_words)
print("\nTop 3 most frequent words:")
print(get_top_words(word_freq))
Output
Word frequency dictionary:
{'the': 2, 'quick': 1, 'brown': 1, 'fox': 1, 'jumps': 1, 'over': 1, 'lazy': 1, 'dog': 2, 'sleeps': 1}
Unique words set:
{'quick', 'brown', 'jumps', 'lazy', 'over', 'fox', 'sleeps', 'dog', 'the'}
Top 3 most frequent words:
[('the', 2), ('dog', 2)]
How it works
The function analyze_text converts the input to lowercase and splits on whitespace, then cleans punctuation from each word. It uses a dictionary word_freq to count occurrences with .get() and a set unique_words to track distinct clean words. The get_top_words helper sorts dictionary items by frequency in descending order and returns the top n pairs. The main block runs a sample text and prints the frequency map, the set of unique words, and the top frequent words.
Common mistakes
- Forgetting to strip punctuation before counting
- Using a list instead of a set for unique words, causing duplicates
- Not using `.get()` to avoid KeyError when incrementing counts
- Ignoring casing, so 'The' and 'the' are counted separately
Variations
- Use `collections.Counter` to count frequencies in one line: `Counter(words)`
- Implement the entire pipeline with list comprehension and `set()` for unique words
Real-world use cases
- Building a word cloud generator that needs frequency analysis of user text.
- Analyzing customer feedback in surveys to identify frequently mentioned topics.
- Preprocessing text for a search engine or NLP pipeline to index unique terms.
Sponsored
More from Dictionaries & sets
- Build a Case-Insensitive Dict with a Wrapper Class in Python medium
- Build a defaultdict histogram of categories in Python easy
- Build adjacency dict graph from edges in Python easy
- Build an OrderedDict insertion order demo in Python 3 easy
- Check Invertible Mapping for Duplicate Values in Python easy
- Compare Two Dictionaries in Python easy
Keep learning
Related tutorials and quizzes for this topic.