How to Normalize Data in Python with Dictionaries and Sets

Normalize a list of dicts by keeping selected keys, stripping/lowercasing strings, and extracting unique sorted values using set comprehension.

Easy Python 3.9+ Aug 9, 2026 Dictionaries & sets 12 views 0 copies

dictionaries sets data-cleaning normalization list-comprehension

Python code

36 lines
Python 3.9+
Run it 
def normalize_data(data, keys):
    """
    Normalize a list of dictionaries by keeping only specified keys
    and converting values to proper types.
    """
    normalized = []
    for item in data:
        clean_item = {}
        for key in keys:
            value = item.get(key)
            if isinstance(value, str):
                value = value.strip().lower()
            clean_item[key] = value
        normalized.append(clean_item)
    return normalized


def unique_values(data, key):
    """Return a sorted set of unique values for a given key."""
    return sorted({item[key] for item in data if item.get(key) is not None})


if __name__ == "__main__":
    raw_data = [
        {"name": "  Alice  ", "age": "25", "city": "New York", "zip": 10001},
        {"name": "BOB", "age": "30", "city": "  boston", "zip": 02110},
        {"name": "  carol ", "age": "22", "city": "New York", "zip": 10001},
        {"name": "DAVID", "age": "35", "city": "Austin", "zip": 73301},
    ]

    clean = normalize_data(raw_data, ["name", "city", "age"])
    print("Normalized data:")
    for row in clean:
        print(row)

    print("\nUnique cities:", unique_values(clean, "city"))

Output

stdout 
Normalized data:
{'name': 'alice', 'city': 'new york', 'age': '25'}
{'name': 'bob', 'city': 'boston', 'age': '30'}
{'name': 'carol', 'city': 'new york', 'age': '22'}
{'name': 'david', 'city': 'austin', 'age': '35'}

Unique cities: ['austin', 'boston', 'new york']

How it works

The normalize_data function iterates over each dictionary and builds a new dictionary containing only the keys specified in keys — this is a classic projection pattern. For each value, it checks if the value is a string; if so, it strips leading/trailing whitespace and lowercases it, making the data consistent for comparisons. The unique_values function uses a set comprehension to collect distinct non‑None values for a key, then returns them sorted — sets guarantee uniqueness and sorting gives a stable, readable order. This pattern keeps the original data unchanged and returns clean copies, which avoids side effects and makes the function reusable.

Common mistakes

  • Forgetting to use `.get()` when accessing keys that may be missing, causing a KeyError.
  • Modifying the original dictionaries in place by assigning directly to the input items.
  • Including None values in the set of unique values because you didn't filter them out.
  • Assuming age should be cast to int, but the example keeps it as a string (conversion logic is only for strings).

Variations

  1. Use a dictionary comprehension inside a list comprehension: `[{k: (v.strip().lower() if isinstance(v, str) else v) for k, v in item.items() if k in keys} for item in data]`.
  2. Use `pandas` to normalize data with `df[keys].apply(lambda col: col.str.strip().str.lower())` for larger datasets.

Real-world use cases

  • Cleaning user-submitted form data before inserting into a database to ensure consistent casing and no extra spaces.
  • Preprocessing JSON payloads from an API to retain only required fields and standardize string values for downstream analysis.
  • Extracting unique category values from a dataset to build filter dropdowns or faceted search options in a web app.

Sponsored

Run this sample

Open the browser IDE to tweak the example and see results without installing anything.

Open editor

More from Dictionaries & sets

Keep learning

Related tutorials and quizzes for this topic.

Tutorials

Quizzes