How to Parse Data with Generators and Comprehensions in Python
This code demonstrates using a generator expression to filter active users and a dictionary comprehension to aggregate scores by name.
Python code
30 linesdef parse_data_helper(raw_records):
"""Extract active users' names and scores from raw records."""
parsed = (
(record["name"], record["score"])
for record in raw_records
if record["active"] and record["score"] >= 0
)
return list(parsed)
def aggregate_scores(parsed_data):
"""Sum scores grouped by name using a comprehension."""
names = {name for name, _ in parsed_data}
return {
name: sum(score for n, score in parsed_data if n == name)
for name in names
}
if __name__ == "__main__":
raw_data = [
{"name": "Ana", "score": 10, "active": True},
{"name": "Bob", "score": 5, "active": False},
{"name": "Ana", "score": 7, "active": True},
{"name": "Cal", "score": -3, "active": True},
]
filtered = parse_data_helper(raw_data)
print("Parsed:", filtered)
print("Aggregated:", aggregate_scores(filtered))
Output
Parsed: [('Ana', 10), ('Ana', 7)]
Aggregated: {'Ana': 17}
How it works
The generator expression inside parse_data_helper lazily filters raw records by checking the active flag and non-negative score, yielding tuples of name and score. list(parsed) materializes the filtered results. aggregate_scores builds a set of unique names and then uses a dictionary comprehension to sum scores for each name, iterating over the list multiple times. This approach showcases readable, functional-style data transformation without explicit loops.
Common mistakes
- Forgetting to wrap the generator expression in `list()` if you need to reuse it later.
- Assuming the generator can be iterated multiple times — it's consumed after one pass.
- Not handling missing dictionary keys, leading to `KeyError` if `'active'` or `'score'` is absent.
Variations
- Use a list comprehension instead of a generator if you don't need lazy evaluation.
- Use `itertools.groupby` on sorted data for more efficient aggregation.
Real-world use cases
- Filtering and extracting user analytics from a JSON API response before storing in a database.
- Aggregating sales totals by product category from a CSV import in an ETL pipeline.
- Preprocessing feature matrices by selecting active records and summarizing per-group metrics.
Sponsored
More from Comprehensions & generators
- Batch Rows in Chunks with a Generator in Python easy
- Build a Generator Pipeline in Python: Filter Then Map medium
- Build a lazy generator to read file lines in Python easy
- Chunk an Iterable into Batches with a Generator in Python easy
- Convert Data in Python with Comprehensions and Generators easy
- Count Data in Python with Comprehensions and Generators easy
Keep learning
Related tutorials and quizzes for this topic.