ML engineering pipelines
Feature prep, batch inference, model-serving hooks, and production ML workflow glue.
Build a Data Helper Class in Python for ML Pipelines
A beginner-friendly Python class that summarizes, filters, and exports ML dataset rows as JSON.
from typing import List, Dict, Any
import json
class DataHelper:
"""Beginner-friendly helpers for ML data pipelines."""
def __init__(self, data: List[Dict[str, Any]]):
self.data = data
self.keys = list(data[0].keys()) if data else []
def summary(self) -> Dict[str, Any]:
"…Create a Minimal Great Expectations Suite Mock in Python
Build a small Python class that mimics a Great Expectations suite, storing and serializing column expectations as JSON.
import json
class GreatExpectationsSuite:
"""A minimal mock of a Great Expectations suite."""
def __init__(self, suite_name, expectations=None):
self.suite_name = suite_name
self.expectations = expectations or []
def add_expectation(self, expectation_type, column=None, kwargs=None):
…Detect Concept Drift in Python with a Simple Statistical Test
Detect concept drift by comparing the mean of recent data against a reference distribution using a z-score-like threshold.
import random
import statistics
def detect_drift(recent, reference, threshold=1.5):
ref_mean = statistics.mean(reference)
ref_std = statistics.stdev(reference)
recent_mean = statistics.mean(recent)
drift_score = abs(recent_mean - ref_mean) / (ref_std if ref_std > 0 else 1)
drifted = drif…How to Build a Data Validation Schema in Python
Create a lightweight validation schema using dataclasses and lambda validators to check fields in a dictionary.
import re
from dataclasses import dataclass, field
from typing import Any, Callable
@dataclass
class Field:
name: str
validator: Callable[[Any], bool]
required: bool = True
def validate(self, value: Any) -> bool:
if not self.required and value is None:
return True
return …How to Build a Mock TFX Pipeline in Python
Simulate a TFX-style ML pipeline with simple Python functions to understand component orchestration, data flow, and artifact passing.
# Mock TFX pipeline to illustrate component orchestration
def CsvExampleGen(data_path):
"""Mock component: Simulates reading CSV data."""
print(f"ExampleGen: Reading from {data_path}")
return {"records": 100, "name": "examples"}
def StatisticsGen(example_artifact):
"""Mock component: Simulates genera…How to Build a Simple ML Pipeline with ZenML in Python
Build a mock machine learning pipeline with ZenML steps for data loading, training, and evaluation, and run it to print the final accuracy.
from zenml import pipeline, step
@step
def load_data() -> dict:
"""Simulate loading data from a source."""
return {"accuracy": 0.0, "loss": 1.0}
@step
def train_model(data: dict) -> dict:
"""Simulate training a model."""
data["accuracy"] = 0.95
data["loss"] = 0.1
return data
@step
def eva…How to Create a Mock Metaflow Flow in Python
Build a minimal Metaflow flow with two sequential steps that pass data between them using instance attributes.
from metaflow import FlowSpec, step, current
class MockFlow(FlowSpec):
"""A minimal Metaflow flow to demonstrate basic steps and branching."""
@step
def start(self):
self.category = "mock"
print(f"Start step for {self.category} flow")
self.next(self.process)
@step
def pr…How to Detect Data Drift with PSI in Python
Calculate the Population Stability Index (PSI) in Python to compare expected vs actual distributions and detect data drift in machine learning pipelines.
import numpy as np
def calculate_psi(expected, actual, buckets=10):
"""Calculate Population Stability Index (PSI) between two distributions."""
# Create bucket edges based on expected distribution percentiles
edges = np.percentile(expected, np.linspace(0, 100, buckets + 1))
edges[-1] = np.inf # Ensur…How to Impute Missing Values with Mean in Python
Replace None values in a list with the mean of the existing values using Python's statistics module.
import statistics
from statistics import mean
def impute_mean(values):
"""Replace None with the mean of the non-None values."""
# Filter out None to compute the mean of existing values
valid = [v for v in values if v is not None]
if not valid:
return values # nothing to impute if all are Non…How to Load CSV Training Data in Python Without Pandas
Load CSV training data using Python's standard library and mock it with io.StringIO for testing, returning headers and rows as dictionaries.
import csv
from pathlib import Path
def load_csv_training_data(file_path: str | Path) -> tuple[list[str], list[dict[str, str]]]:
"""Load CSV training data and return headers plus rows as dictionaries."""
with open(file_path, mode="r", newline="", encoding="utf-8") as csv_file:
reader = csv.DictReader…How to Load, Save, and Split JSON Data in Python
Provides helper functions to load, save, and split JSON dictionary data for simple ML pipeline preprocessing.
import json
from pathlib import Path
def load_json_data(file_path):
"""Load JSON data from a file, returning an empty dict if missing."""
path = Path(file_path)
if path.exists():
with path.open("r", encoding="utf-8") as f:
return json.load(f)
return {}
def save_json_data(data, f…How to Mock Kedro Pipeline Nodes in Python
Create a modular Kedro pipeline with node functions, namespacing, and input/output mapping to mock pipeline execution locally.
from kedro.pipeline import Pipeline, node
from kedro.pipeline.modular_pipeline import pipeline as modular_pipeline
def preprocess(data: list) -> list:
"""Clean data by removing None values."""
return [item for item in data if item is not None]
def transform(data: list) -> list:
"""Add 1 to each numeric…How to Mock a Kubeflow Pipeline in Python
Build a minimal in-memory mock of a Kubeflow pipeline DAG using dataclasses and OrderedDict to chain component functions.
from typing import Dict, Any
from dataclasses import dataclass, field
from collections import OrderedDict
@dataclass
class KubeflowPipelineMock:
"""A minimal mock of a Kubeflow pipeline DAG."""
name: str
components: OrderedDict[str, callable] = field(default_factory=OrderedDict)
def add_component(se…How to Train a Gradient Boosting Regressor in Python
Build and evaluate a scikit-learn GradientBoostingRegressor on a synthetic dataset, printing test MSE and feature importances.
import numpy as np
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.metrics import mean_squared_error
def train_gradient_boosting_mock():
# Toy regression dataset
np.random.seed(42)
X = np.random.rand(100, 3) * 10
y = 2 * X[:, 0] - 1.5 * X[:, 1] + 0.5 * X[:, 2] + np.random.normal(0,…How to do feature selection with VarianceThreshold in Python
This code demonstrates how to use scikit-learn's VarianceThreshold to remove low-variance features from a NumPy array, keeping only those that vary enough to be useful for modeling.
import numpy as np
from sklearn.feature_selection import VarianceThreshold
def main():
# Mock dataset: 4 samples, 5 features
X = np.array([
[0.1, 0.2, 1.0, 1.0, 0.5],
[0.2, 0.2, 0.0, 1.0, 0.4],
[0.1, 0.2, 1.0, 1.0, 0.6],
[0.3, 0.2, 1.0, 0.0, 0.5]
])
# Select features w…How to mock an artifact store with local paths in Python for ML pipelines
Create a temporary local artifact store with dummy files and metadata to test ML pipeline code without real storage.
import tempfile
from pathlib import Path
import json
def create_artifact_store_mock(base_path: Path = None):
"""Create a local artifact store mock directory structure."""
if base_path is None:
base_path = Path(tempfile.mkdtemp())
store_layout = {
"artifacts": [
{"name": "mode…How to ordinal encode categorical data in Python with sklearn
Convert job title categories into ordinal numeric labels using sklearn's OrdinalEncoder with explicit ordering.
from sklearn.preprocessing import OrdinalEncoder
import numpy as np
# Mock data: small job title categories with known ordering
data = np.array([
["intern"],
["junior"],
["mid"],
["senior"],
["lead"]
])
# Define the ordinal order (lowest to highest)
categories = [["intern", "junior", "mid", "seni…K-Fold Cross Validation in Python: A Simple Implementation
Implements k-fold cross validation from scratch, splitting data into folds and computing MSE scores for a baseline mean-predictor model.
import random
from statistics import mean
def cross_validation_scores(data, labels, k=5, seed=42):
random.seed(seed)
indices = list(range(len(data)))
random.shuffle(indices)
fold_size = len(indices) // k
folds = []
for i in range(k):
if i == k - 1:
folds.append(indices[i *…Load CSV Training Data Without Pandas in Python
This code loads a CSV file into a list of dictionaries using only the standard library, ideal for small ML training data without heavy dependencies.
import csv
from pathlib import Path
def load_csv(path):
"""Load CSV file into list of dicts without pandas."""
rows = []
with open(path, newline='', encoding='utf-8') as f:
reader = csv.DictReader(f)
for row in reader:
rows.append(dict(row))
return rows
if __name__ == "__m…Mock a Flyte ML workflow in Python
Build a lightweight mock of a Flyte ML pipeline with dataclasses and a simple execution loop that passes outputs between tasks.
from dataclasses import dataclass, field
from typing import List, Dict, Optional
import time
@dataclass
class FlyteTask:
name: str
inputs: Dict = field(default_factory=dict)
outputs: Dict = field(default_factory=dict)
def run(self) -> Dict:
time.sleep(0.1) # simulate work
return sel…Model registry version mock in Python
A simple in-memory model registry that stores model versions with metadata and supports version listing and latest retrieval.
class ModelRegistry:
def __init__(self):
self.models = {}
def register(self, name, version, model_type, metrics=None):
if name not in self.models:
self.models[name] = []
entry = {
"version": version,
"model_type": model_type,
"metrics": m…One Hot Encode Categories in Python
Convert a list of categorical strings into one-hot encoded numeric vectors using pure Python and NumPy.
import numpy as np
categories = ["red", "green", "blue", "red", "blue", "green", "red"]
unique = sorted(set(categories))
lookup = {cat: i for i, cat in enumerate(unique)}
one_hot = []
for cat in categories:
row = [0] * len(unique)
row[lookup[cat]] = 1
one_hot.append(row)
print("Categories:", categories…Train Logistic Regression From Scratch in Python
Trains a binary logistic regression model using gradient descent on mock data, printing learned weights and probabilities.
import numpy as np
# Mock data: 2 features, binary classification
X = np.array([[1, 2], [2, 3], [3, 4], [4, 5], [5, 6]])
y = np.array([0, 0, 1, 1, 1])
# Add bias term (column of ones)
X_b = np.c_[np.ones((X.shape[0], 1)), X]
# Initialize parameters
theta = np.zeros(X_b.shape[1])
# Hyperparameters
learning_rate = 0…Browse by section
Each section groups closely related Python snippets.
ML engineering pipelines — Python code examples
What you will find here
This page collects ml engineering pipelines snippets — short, copy-ready Python you can paste into our free online IDE and run without installing anything. Each sample includes a plain-English explanation and the full source code.
Samples vs tutorials and challenges
Samples are quick reference — one concept per page. For step-by-step teaching, use our Python tutorials. To test yourself, try quizzes or coding challenges. Clean up style with the Python formatter.