How to Convert Data Types in a Python Data Pipeline

Demonstrates a simple Python data pipeline that converts string values to proper types (bool, int, float, datetime) and outputs structured JSON.

Easy Python 3.9+ Aug 9, 2026 Data pipelines & processing 12 views 0 copies

Python code

42 lines
Python 3.9+
import json
from datetime import datetime

def convert_value(value):
    """Convert string values to appropriate Python types."""
    if value.lower() == "true":
        return True
    if value.lower() == "false":
        return False
    if value.isdigit():
        return int(value)
    try:
        return float(value)
    except ValueError:
        pass
    try:
        return datetime.fromisoformat(value)
    except ValueError:
        return value

def process_pipeline(raw_data):
    """Convert and structure raw CSV-like data into JSON-friendly format."""
    records = []
    for line in raw_data.strip().split("\n"):
        fields = [item.strip() for item in line.split(",")]
        record = {
            "name": fields[0],
            "age": convert_value(fields[1]),
            "active": convert_value(fields[2]),
            "score": convert_value(fields[3]),
            "joined": convert_value(fields[4]),
        }
        records.append(record)
    return json.dumps(records, indent=2, default=str)

if __name__ == "__main__":
    raw_data = """
Alice, 30, true, 92.5, 2023-05-15
Bob, 25, false, 78.0, 2022-11-01
Charlie, 35, true, 88.75, 2024-01-20
"""
    print(process_pipeline(raw_data))

Output

stdout
[
  {
    "name": "Alice",
    "age": 30,
    "active": true,
    "score": 92.5,
    "joined": "2023-05-15"
  },
  {
    "name": "Bob",
    "age": 25,
    "active": false,
    "score": 78.0,
    "joined": "2022-11-01"
  },
  {
    "name": "Charlie",
    "age": 35,
    "active": true,
    "score": 88.75,
    "joined": "2024-01-20"
  }
]

How it works

The convert_value function checks values in order: booleans, integers, floats, then datetimes, falling back to the original string. datetime.fromisoformat parses ISO 8601 dates, but the output shows them as strings because json.dumps with default=str converts datetime objects to their ISO representation. The process_pipeline function splits each line on commas, strips whitespace, and builds a dictionary per record. Using json.dumps with indent=2 gives a readable JSON output, making the pipeline easy to inspect and debug.

Common mistakes

  • Forgetting to strip whitespace from fields, leading to 'true ' failing the boolean check
  • Assuming all values are strings and not handling conversion errors gracefully
  • Using `json.dumps` without `default=str`, causing TypeError on datetime objects
  • Not handling empty lines in raw data, which can produce empty records

Variations

  1. Use `csv.DictReader` for more robust CSV parsing with headers
  2. Add `encoding='utf-8'` when reading from a file to handle non-ASCII characters

Real-world use cases

  • Transforming exported CSV exports from legacy systems into JSON payloads for a new API.
  • Normalizing data from multiple source files before loading into a database or data warehouse.
  • Preparing user profile data from a form submission to store in a structured format for analytics.

Sponsored

Run this sample

Open the browser IDE to tweak the example and see results without installing anything.

Open editor

More from Data pipelines & processing

Related tutorials and quizzes for this topic.