How to Handle Missing Values in a CSV Numeric Column in Python
Clean missing entries in a CSV numeric column by filling them with the mean, median, a custom value, or dropping rows.
Python code
46 linesimport csv
from pathlib import Path
import statistics
def clean_csv_numeric(input_path: str, output_path: str, column: str, strategy: str = "mean") -> None:
"""
Handles missing values in a numeric column of a CSV file.
Strategies: 'mean', 'median', 'drop', or 'fill' with a specified value.
"""
rows = list(csv.DictReader(Path(input_path).open(newline="")))
values = []
for row in rows:
if row[column] not in ("", None):
try:
values.append(float(row[column]))
except ValueError:
pass
if strategy == "mean":
fill_value = statistics.mean(values) if values else 0.0
elif strategy == "median":
fill_value = statistics.median(values) if values else 0.0
elif strategy == "drop":
rows = [r for r in rows if r[column] not in ("", None)]
fill_value = None
else: # custom fill value passed as strategy (e.g., "0" or "10")
fill_value = float(strategy)
if fill_value is not None:
for row in rows:
if row[column] in ("", None):
row[column] = f"{fill_value:.2f}"
fieldnames = rows[0].keys() if rows else []
with Path(output_path).open("w", newline="") as f:
writer = csv.DictWriter(f, fieldnames=fieldnames)
writer.writeheader()
writer.writerows(rows)
if __name__ == "__main__":
sample = "sample_data.csv"
with Path(sample).open("w", newline="") as f:
f.write("name,age,salary\nAlice,30,50000\nBob,,60000\nCharlie,25,\nDiana,35,70000\n")
clean_csv_numeric(sample, "cleaned.csv", column="salary", strategy="mean")
print(Path("cleaned.csv").read_text())
Output
name,age,salary
Alice,30,50000.00
Bob,,60000.00
Charlie,25,60000.00
Diana,35,70000.00
How it works
The function clean_csv_numeric reads the CSV with csv.DictReader, which represents each row as a dictionary keyed by column names. It first collects all non-empty, numeric values from the target column (ignoring blanks and non-parseable strings) to compute the fill statistic. Depending on the chosen strategy, it either fills missing cells with a formatted float (two decimal places) or removes rows that have missing values. Writing with csv.DictWriter preserves the header and column order, and the fieldnames are taken from the first row's keys. The stdlib modules csv, pathlib, and statistics handle everything without third-party dependencies.
Common mistakes
- Comparing `row[column]` to `None` when CSV fields are always strings, so blanks are `''` not `None`.
- Forgetting to convert the custom fill value (a string) to float before assigning to the cell.
- Assuming all values in the column parse as floats, causing silent skips when using `try/except`.
Variations
- Use `pandas.DataFrame.fillna(method='ffill')` for a more feature-rich alternative with `pip install pandas`.
- Call `csv.DictReader` directly on an open file object instead of wrapping with `Path.open()`.
Real-world use cases
- Preprocessing sensor logs where occasional dropped readings leave blank cells before feeding data into a regression model.
- Cleaning exported financial reports that contain missing revenue values so totals and averages become accurate.
- Standardizing user-uploaded spreadsheets in a web app to fill gaps in age or salary columns before processing.
Sponsored
More from Files & data
- Append a Line to a Log File in Python easy
- Audit File Permissions Across a Project in Python easy
- Automatically Detect Corrupted Files Using SHA-256 Checksums in Python easy
- Automatically Highlight Data Validation Errors Inside Excel Files in Python easy
- Build a Command-Line To-Do List Application with Data Persistence in Python easy
- Build a File Index by Relative Path Hash Map in Python easy
Keep learning
Related tutorials and quizzes for this topic.