Reference library

Data pipelines & processing

ETL-style flows, batch transforms, validation, and moving data between formats.

5 matches

Build a Python Utility That Detects Duplicate Records Across Multiple Excel Sheets

A Python utility that uses pandas to find overlapping records across different Excel sheets based on specified key columns.

pandas excel data cleaning

Python

import pandas as pd
from pathlib import Path

def find_duplicate_records_across_sheets(file_path: str, key_columns: list, sheet_names: list) -> dict:
    """
    Detect duplicate records across multiple Excel sheets based on specified key columns.
    
    Args:
        file_path: Path to the Excel file
        key_co…

46 0 Open

Data pipelines & processing easy

How to Explode an Array Field into Multiple Rows in Python

This code flattens a list of dictionaries by exploding each array field value into its own row, duplicating the other fields as needed.

data transformation arrays flattening

Python

from collections import defaultdict

data = [
    {"id": 1, "name": "Alice", "tags": ["python", "data", "ai"]},
    {"id": 2, "name": "Bob", "tags": ["web", "devops"]},
    {"id": 3, "name": "Carol", "tags": []},
]

def explode_array_field(records, array_field):
    result = []
    for record in records:
        for v…

11 0 Open

Data pipelines & processing medium

How to Find Missing Values in Large Datasets in Python

Analyze missing values across multiple large pandas DataFrames with counts and percentages.

pandas missing-data data-cleaning

Python

import pandas as pd
import numpy as np

def find_missing_values_summary(datasets):
    """Analyze missing values across multiple datasets (dict of name: DataFrame)."""
    summary = {}
    for name, df in datasets.items():
        missing_count = df.isnull().sum()
        total_rows = len(df)
        missing_pct = (mi…

41 0 Open

Data pipelines & processing easy

How to Unpivot Wide to Long with pandas melt in Python

This code demonstrates how to use pandas.melt to unpivot a wide DataFrame into a tidy long format, converting subject columns into rows.

pandas melt reshape

Python

import pandas as pd

# Sample wide-format data
df_wide = pd.DataFrame({
    'id': [1, 2, 3],
    'name': ['Alice', 'Bob', 'Charlie'],
    'math': [90, 85, 95],
    'science': [80, 92, 88]
})

print("Original wide DataFrame:")
print(df_wide)

# Melt: unpivot subject columns into rows
df_long = pd.melt(
    df_wide,
   …

15 0 Open

Data pipelines & processing easy

Union Multiple DataFrames with Aligned Columns in Python

Concatenate DataFrames with different columns, aligning them and filling missing values with NaN using pandas concat.

pandas dataframes concat

Python

import pandas as pd
from io import StringIO

# Sample dataframes with different columns
df1 = pd.DataFrame({
    'id': [1, 2, 3],
    'name': ['Alice', 'Bob', 'Charlie'],
    'age': [25, 30, 35]
})

df2 = pd.DataFrame({
    'id': [4, 5],
    'name': ['Diana', 'Eve'],
    'city': ['NYC', 'LA']
})

df3 = pd.DataFrame({
…

14 0 Open

Browse by section

Each section groups closely related Python snippets.