Normalize Data in Python with Comprehensions and Generators
Clean a list by dropping None values with a comprehension, then min-max normalize it using a lazy generator expression — a beginner-friendly data preparation pattern.
Python code
31 linesimport statistics
# Sample raw data including missing and outlier-ish values
raw = [22, 18, None, 25, 30, 19, 22, 17, None, 28, 24]
# Clean the data: drop None values using a list comprehension
clean = [x for x in raw if x is not None]
# Normalize using min-max scaling with a generator expression
min_val = min(clean)
max_val = max(clean)
def scale(value):
return (value - min_val) / (max_val - min_val)
# Generator expression: lazy evaluation, yields values one at a time
normalized_gen = (scale(x) for x in clean)
# Convert generator to list to observe results
normalized = list(normalized_gen)
# Show stats and output
print("Raw:", raw)
print("Clean:", clean)
print("Normalized (rounded to 3 decimals):", [round(x, 3) for x in normalized])
print("Mean:", round(statistics.mean(normalized), 3))
print("Min:", round(min(normalized), 3), "Max:", round(max(normalized), 3))
# Demonstrate generator laziness: sum without materializing
total = sum(scale(x) for x in clean)
print("Sum (from generator):", round(total, 3))
Output
Raw: [22, 18, None, 25, 30, 19, 22, 17, None, 28, 24]
Clean: [22, 18, 25, 30, 19, 22, 17, 28, 24]
Normalized (rounded to 3 decimals): [0.385, 0.077, 0.615, 1.0, 0.154, 0.385, 0.0, 0.846, 0.538]
Mean: 0.444
Min: 0.0 Max: 1.0
Sum (from generator): 4.0
How it works
The list comprehension [x for x in raw if x is not None] creates a new list with only valid values, making the dataset ready for numerical processing. The generator expression (scale(x) for x in clean) computes normalized values lazily — each value is produced on demand rather than all at once, saving memory for large datasets. The closure over min_val and max_val in scale keeps the normalization logic simple and reusable. Converting to a list with list(normalized_gen) materializes results for inspection, while sum(scale(x) for x in clean) shows how generators avoid intermediate storage.
Common mistakes
- Using `== None` instead of `is None` — identity checks are the Pythonic way for None.
- Modifying the raw list instead of creating a new cleaned one, which changes source data.
Variations
- Use `(x for x in raw if x is not None)` as a generator directly without materializing the clean list.
- Replace min-max scaling with `(x - mean) / std` for z-score normalization using statistics.
Real-world use cases
- Preparing feature columns for machine learning models by cleaning and scaling numeric data.
- Filtering faulty sensor readings (None values) before calculating rolling averages in IoT pipelines.
- Normalizing API response values to a 0–1 range for consistent dashboard chart scaling.
Sponsored
More from Comprehensions & generators
- Batch Rows in Chunks with a Generator in Python easy
- Build a Generator Pipeline in Python: Filter Then Map medium
- Build a lazy generator to read file lines in Python easy
- Chunk an Iterable into Batches with a Generator in Python easy
- Convert Data in Python with Comprehensions and Generators easy
- Count Data in Python with Comprehensions and Generators easy
Keep learning
Related tutorials and quizzes for this topic.