How to Convert Data Types in a Python Data Pipeline
Demonstrates a simple Python data pipeline that converts string values to proper types (bool, int, float, datetime) and outputs structured JSON.
Python code
42 linesimport json
from datetime import datetime
def convert_value(value):
"""Convert string values to appropriate Python types."""
if value.lower() == "true":
return True
if value.lower() == "false":
return False
if value.isdigit():
return int(value)
try:
return float(value)
except ValueError:
pass
try:
return datetime.fromisoformat(value)
except ValueError:
return value
def process_pipeline(raw_data):
"""Convert and structure raw CSV-like data into JSON-friendly format."""
records = []
for line in raw_data.strip().split("\n"):
fields = [item.strip() for item in line.split(",")]
record = {
"name": fields[0],
"age": convert_value(fields[1]),
"active": convert_value(fields[2]),
"score": convert_value(fields[3]),
"joined": convert_value(fields[4]),
}
records.append(record)
return json.dumps(records, indent=2, default=str)
if __name__ == "__main__":
raw_data = """
Alice, 30, true, 92.5, 2023-05-15
Bob, 25, false, 78.0, 2022-11-01
Charlie, 35, true, 88.75, 2024-01-20
"""
print(process_pipeline(raw_data))
Output
[
{
"name": "Alice",
"age": 30,
"active": true,
"score": 92.5,
"joined": "2023-05-15"
},
{
"name": "Bob",
"age": 25,
"active": false,
"score": 78.0,
"joined": "2022-11-01"
},
{
"name": "Charlie",
"age": 35,
"active": true,
"score": 88.75,
"joined": "2024-01-20"
}
]
How it works
The convert_value function checks values in order: booleans, integers, floats, then datetimes, falling back to the original string. datetime.fromisoformat parses ISO 8601 dates, but the output shows them as strings because json.dumps with default=str converts datetime objects to their ISO representation. The process_pipeline function splits each line on commas, strips whitespace, and builds a dictionary per record. Using json.dumps with indent=2 gives a readable JSON output, making the pipeline easy to inspect and debug.
Common mistakes
- Forgetting to strip whitespace from fields, leading to 'true ' failing the boolean check
- Assuming all values are strings and not handling conversion errors gracefully
- Using `json.dumps` without `default=str`, causing TypeError on datetime objects
- Not handling empty lines in raw data, which can produce empty records
Variations
- Use `csv.DictReader` for more robust CSV parsing with headers
- Add `encoding='utf-8'` when reading from a file to handle non-ASCII characters
Real-world use cases
- Transforming exported CSV exports from legacy systems into JSON payloads for a new API.
- Normalizing data from multiple source files before loading into a database or data warehouse.
- Preparing user profile data from a form submission to store in a structured format for analytics.
Sponsored
More from Data pipelines & processing
- Add a UUID Surrogate Key to Each Row in a CSV with Python easy
- Attach Source File Metadata to Records in Python easy
- Build a Python Utility That Detects Duplicate Records Across Multiple Excel Sheets medium
- Check Null Rate Threshold in PySpark DataFrame medium
- Count Records Processed per Category in Python easy
- Create Data Helper Functions in Python for Beginners easy
Keep learning
Related tutorials and quizzes for this topic.