How to Replace Outliers Beyond Threshold with Cap in Python
Replace values that fall below a lower threshold or above an upper threshold by capping them to the threshold values using a simple Python function.
Python code
22 linesdef replace_outliers_with_cap(data, lower_threshold=None, upper_threshold=None):
"""Replace values beyond given thresholds with the threshold values (capping)."""
if lower_threshold is None and upper_threshold is None:
raise ValueError("At least one threshold must be provided.")
capped_data = list(data)
if upper_threshold is not None:
capped_data = [min(value, upper_threshold) for value in capped_data]
if lower_threshold is not None:
capped_data = [max(value, lower_threshold) for value in capped_data]
return capped_data
if __name__ == "__main__":
original_data = [5, 12, 18, 25, 9, 31, 7, 40, 3, 15]
result = replace_outliers_with_cap(original_data, lower_threshold=4, upper_threshold=30)
print("Original: ", original_data)
print("Capped: ", result)
Output
Original: [5, 12, 18, 25, 9, 31, 7, 40, 3, 15]
Capped: [5, 12, 18, 25, 9, 30, 7, 30, 4, 15]
How it works
The function first creates a copy of the input data to avoid mutating the original list. It applies the upper threshold with min(value, upper_threshold), which clamps any value above the threshold down to the threshold. Similarly, the lower threshold uses max(value, lower_threshold) to raise values below it. The order of operations matters: applying the upper clamp first, then the lower clamp, ensures that values are correctly bounded within the desired range even if thresholds overlap. List comprehensions make the transformation concise and efficient.
Common mistakes
- Mutating the input list in place instead of returning a new list.
- Forgetting to handle the case where both thresholds are None, leading to unexpected behavior.
- Applying the lower threshold before the upper threshold can cause incorrect results if thresholds are inverted.
- Not checking that the lower threshold is less than the upper threshold when both are provided.
Variations
- Use `numpy.clip` for numpy arrays: `np.clip(data, lower_threshold, upper_threshold)`.
- Add a `replace_with` parameter to replace outliers with a specific value (e.g., median) instead of the threshold.
Real-world use cases
- Preprocessing sensor readings to remove spikes before feeding into a machine learning model.
- Capping extreme values in financial data to prevent skewed statistical analysis and reporting.
- Sanitizing user input in a web app to enforce bounds on numeric parameters like age or price.
Sponsored
More from Algorithms & data structures
- Binary Search for Ship Capacity in Python medium
- Binary Search on Answer in Python: Koko Eating Bananas medium
- Bucket Numbers into Histogram Bin Counts in Python easy
- Container With Most Water: Two-Pointer Solution in Python medium
- Count Smaller Elements to the Right in Python easy
- Depth First Search Traversal Order in Python easy
Keep learning
Related tutorials and quizzes for this topic.