Reference library

Data pipelines & processing

ETL-style flows, batch transforms, validation, and moving data between formats.

4 matches

Build a Python Utility That Detects Duplicate Records Across Multiple Excel Sheets

A Python utility that uses pandas to find overlapping records across different Excel sheets based on specified key columns.

pandas excel data cleaning

Python

import pandas as pd
from pathlib import Path

def find_duplicate_records_across_sheets(file_path: str, key_columns: list, sheet_names: list) -> dict:
    """
    Detect duplicate records across multiple Excel sheets based on specified key columns.
    
    Args:
        file_path: Path to the Excel file
        key_co…

47 0 Open

Data pipelines & processing medium

Check Null Rate Threshold in PySpark DataFrame

This PySpark code checks the null rate of specified DataFrame columns against a threshold and returns violations.

pyspark data quality null check

Python

from pyspark.sql import SparkSession
from pyspark.sql.functions import col, sum, count

def check_null_rate(df, threshold=0.2, columns=None):
    """
    Check null rate for specified columns (or all) against a threshold.
    Returns columns that exceed the threshold.
    """
    cols = columns or df.columns
    total…

16 0 Open

Data pipelines & processing easy

How to Unpivot Wide to Long with pandas melt in Python

This code demonstrates how to use pandas.melt to unpivot a wide DataFrame into a tidy long format, converting subject columns into rows.

pandas melt reshape

Python

import pandas as pd

# Sample wide-format data
df_wide = pd.DataFrame({
    'id': [1, 2, 3],
    'name': ['Alice', 'Bob', 'Charlie'],
    'math': [90, 85, 95],
    'science': [80, 92, 88]
})

print("Original wide DataFrame:")
print(df_wide)

# Melt: unpivot subject columns into rows
df_long = pd.melt(
    df_wide,
   …

15 0 Open

Data pipelines & processing easy

Union Multiple DataFrames with Aligned Columns in Python

Concatenate DataFrames with different columns, aligning them and filling missing values with NaN using pandas concat.

pandas dataframes concat

Python

import pandas as pd
from io import StringIO

# Sample dataframes with different columns
df1 = pd.DataFrame({
    'id': [1, 2, 3],
    'name': ['Alice', 'Bob', 'Charlie'],
    'age': [25, 30, 35]
})

df2 = pd.DataFrame({
    'id': [4, 5],
    'name': ['Diana', 'Eve'],
    'city': ['NYC', 'LA']
})

df3 = pd.DataFrame({
…

14 0 Open

Browse by section

Each section groups closely related Python snippets.