ML engineering pipelines
Feature prep, batch inference, model-serving hooks, and production ML workflow glue.
Compare Model A vs Model B Metrics in Python
A script that simulates and compares metrics between two ML models, showing a formatted diff table for quick insight.
import random
def compare_a_b(samples=5):
"""Mock comparison of model A vs model B predictions."""
metrics = ["accuracy", "precision", "recall", "f1"]
print(f"{'Metric':<12}{'Model A':>10}{'Model B':>10}{'Diff':>10}")
print("-" * 42)
random.seed(42)
for metric in metrics:
a = round(r…How to Build a Data Validation Schema in Python
Create a lightweight validation schema using dataclasses and lambda validators to check fields in a dictionary.
import re
from dataclasses import dataclass, field
from typing import Any, Callable
@dataclass
class Field:
name: str
validator: Callable[[Any], bool]
required: bool = True
def validate(self, value: Any) -> bool:
if not self.required and value is None:
return True
return …How to Build a Mock ML Pipeline with Prefect in Python
Create a lightweight Prefect flow with mock preprocessing, training, and evaluation tasks to prototype an ML pipeline end-to-end.
from prefect import task, flow
from datetime import datetime
@task
def preprocess_data(raw_value: float) -> float:
"""Mock preprocessing: normalize the input value."""
return raw_value / 100.0
@task
def train_model(features: float) -> dict:
"""Mock training: return a fake model artifact."""
return …How to Build a Mock Offline Feature Store in Python
Build an in-memory mock of an offline feature store with a dict-based FeatureStore class for storing and retrieving ML features by entity ID.
from datetime import datetime
from collections import defaultdict
class FeatureStore:
"""Simple in-memory mock of an offline feature store."""
def __init__(self):
self._features = defaultdict(dict)
def ingest(self, entity_id, feature_name, value, timestamp=None):
ts = timestamp or datet…How to Build a Mock TFX Pipeline in Python
Simulate a TFX-style ML pipeline with simple Python functions to understand component orchestration, data flow, and artifact passing.
# Mock TFX pipeline to illustrate component orchestration
def CsvExampleGen(data_path):
"""Mock component: Simulates reading CSV data."""
print(f"ExampleGen: Reading from {data_path}")
return {"records": 100, "name": "examples"}
def StatisticsGen(example_artifact):
"""Mock component: Simulates genera…How to Build a Simple ML Pipeline with ZenML in Python
Build a mock machine learning pipeline with ZenML steps for data loading, training, and evaluation, and run it to print the final accuracy.
from zenml import pipeline, step
@step
def load_data() -> dict:
"""Simulate loading data from a source."""
return {"accuracy": 0.0, "loss": 1.0}
@step
def train_model(data: dict) -> dict:
"""Simulate training a model."""
data["accuracy"] = 0.95
data["loss"] = 0.1
return data
@step
def eva…How to Build an sklearn Pipeline with ColumnTransformer in Python
A mock example showing how to chain preprocessing and a regression model into a single sklearn Pipeline, scaling numeric features and one-hot encoding categorical features with ColumnTransformer.
import numpy as np
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.pipeline import Pipeline
from sklearn.linear_model import LinearRegression
# Mock dataset
X = np.array([[1, 'red'], [2, 'blue'], [3, 'red'], [4, 'green'], [5, 'blue']], dtype=o…How to Compute a Confusion Matrix in Python
Compute a multi-class confusion matrix from true and predicted labels using pure Python dictionaries and nested lists, then format it for readable output.
from collections import defaultdict
def compute_confusion_matrix(y_true, y_pred, labels):
"""Compute confusion matrix using Python dicts and nested lists."""
label_index = {label: i for i, label in enumerate(labels)}
matrix = [[0] * len(labels) for _ in range(len(labels))]
for true, pred in zip(y…How to Create a Mock Metaflow Flow in Python
Build a minimal Metaflow flow with two sequential steps that pass data between them using instance attributes.
from metaflow import FlowSpec, step, current
class MockFlow(FlowSpec):
"""A minimal Metaflow flow to demonstrate basic steps and branching."""
@step
def start(self):
self.category = "mock"
print(f"Start step for {self.category} flow")
self.next(self.process)
@step
def pr…How to Create a Mock ONNX Model in Python
Build and export a minimal mock ONNX model with a Reshape and Gemm layer using the onnx helper API.
import onnx
import numpy as np
from onnx import helper, TensorProto
def create_mock_model():
# Define input and output tensors
input_tensor = helper.make_tensor_value_info('input', TensorProto.FLOAT, [1, 3, 224, 224])
output_tensor = helper.make_tensor_value_info('output', TensorProto.FLOAT, [1, 10])
…How to Define Dagster ML Assets in Python
Define a chain of Dagster software-defined assets that compute raw features, normalized features, and predictions for an ML pipeline.
from dagster import asset
@asset
def raw_features():
return {"sepal_length": [5.1, 4.9, 6.2], "sepal_width": [3.5, 3.0, 3.4]}
@asset
def normalized_features(raw_features):
values = raw_features["sepal_length"]
mean = sum(values) / len(values)
std = (sum((x - mean) ** 2 for x in values) / len(values…How to Detect Data Drift with PSI in Python
Calculate the Population Stability Index (PSI) in Python to compare expected vs actual distributions and detect data drift in machine learning pipelines.
import numpy as np
def calculate_psi(expected, actual, buckets=10):
"""Calculate Population Stability Index (PSI) between two distributions."""
# Create bucket edges based on expected distribution percentiles
edges = np.percentile(expected, np.linspace(0, 100, buckets + 1))
edges[-1] = np.inf # Ensur…How to Do Random Search for Hyperparameter Tuning in Python
A mock random search that samples hyperparameter combinations from a grid and ranks them by a dummy score, with a reproducible seed.
import random
# Mock random search over a small hyperparameter grid
param_grid = {
"learning_rate": [0.001, 0.01, 0.1],
"batch_size": [16, 32, 64],
"num_layers": [1, 2, 3]
}
def random_search(grid, n_iter=5, seed=42):
"""Perform random search over a hyperparameter grid."""
random.seed(seed)
k…How to Evaluate Accuracy, Precision, and Recall in Python
Compute accuracy, precision, and recall for a binary classification model using scikit-learn's metrics functions.
from sklearn.metrics import accuracy_score, precision_score, recall_score
if __name__ == "__main__":
y_true = [0, 1, 1, 0, 1, 0, 1, 1]
y_pred = [0, 1, 0, 0, 1, 0, 1, 1]
accuracy = accuracy_score(y_true, y_pred)
precision = precision_score(y_true, y_pred)
recall = recall_score(y_true, y_pred)
…How to Generate Experiment Tracking Run IDs in Python
Generate unique experiment run IDs with timestamps and random suffixes for tracking ML pipeline executions.
import random
import string
import time
def generate_run_id(prefix="exp"):
timestamp = time.strftime("%Y%m%d_%H%M%S")
suffix = "".join(random.choices(string.ascii_lowercase + string.digits, k=6))
return f"{prefix}_{timestamp}_{suffix}"
if __name__ == "__main__":
# Simulate tracking three experiment r…How to Impute Missing Values with Mean in Python
Replace None values in a list with the mean of the existing values using Python's statistics module.
import statistics
from statistics import mean
def impute_mean(values):
"""Replace None with the mean of the non-None values."""
# Filter out None to compute the mean of existing values
valid = [v for v in values if v is not None]
if not valid:
return values # nothing to impute if all are Non…How to Load CSV Training Data in Python Without Pandas
Load CSV training data using Python's standard library and mock it with io.StringIO for testing, returning headers and rows as dictionaries.
import csv
from pathlib import Path
def load_csv_training_data(file_path: str | Path) -> tuple[list[str], list[dict[str, str]]]:
"""Load CSV training data and return headers plus rows as dictionaries."""
with open(file_path, mode="r", newline="", encoding="utf-8") as csv_file:
reader = csv.DictReader…How to Load, Save, and Split JSON Data in Python
Provides helper functions to load, save, and split JSON dictionary data for simple ML pipeline preprocessing.
import json
from pathlib import Path
def load_json_data(file_path):
"""Load JSON data from a file, returning an empty dict if missing."""
path = Path(file_path)
if path.exists():
with path.open("r", encoding="utf-8") as f:
return json.load(f)
return {}
def save_json_data(data, f…How to Mock Cron Schedule in Python
Compute the next scheduled run time for a cron expression using a pure-Python mock parser.
import re
from datetime import datetime, timedelta
class CronMock:
def __init__(self, expression):
self.expression = expression
self.minutes = self._parse_field(expression.split()[0], 0, 59)
self.hours = self._parse_field(expression.split()[1], 0, 23)
self.days = self._parse_field(…How to Mock Kedro Pipeline Nodes in Python
Create a modular Kedro pipeline with node functions, namespacing, and input/output mapping to mock pipeline execution locally.
from kedro.pipeline import Pipeline, node
from kedro.pipeline.modular_pipeline import pipeline as modular_pipeline
def preprocess(data: list) -> list:
"""Clean data by removing None values."""
return [item for item in data if item is not None]
def transform(data: list) -> list:
"""Add 1 to each numeric…How to Mock MLflow Model Registration in Python
Build a lightweight in-memory mock of MLflow's MlflowClient to test model registration, versioning, and stage transitions without a tracking server.
from mlflow.tracking import MlflowClient
from mlflow.entities import ModelVersion, Model
class MockMlflowClient:
"""Minimal mock of MlflowClient's model registration methods."""
def __init__(self):
self.registered_models = {}
self.model_versions = {}
def register_model(self, mod…How to Mock MLflow log_params and log_metrics in Python
Use unittest.mock to patch MLflow's log_param and log_metric, run the training function, and verify logging calls without touching a real tracking server.
from unittest.mock import Mock, patch
import mlflow
def train_model():
mlflow.log_param("learning_rate", 0.01)
mlflow.log_param("epochs", 10)
mlflow.log_metric("accuracy", 0.95)
mlflow.log_metric("loss", 0.05)
return "Training completed"
if __name__ == "__main__":
with patch("mlflow.log_par…How to Mock ROC AUC in Python
Compute ROC AUC from scratch in Python using pairwise comparisons between positive and negative score distributions, ideal for testing ML models without sklearn.
import random
from math import comb
def mock_roc_auc(scores, labels):
"""Compute mock ROC AUC by simulating a classifier's score distribution."""
random.seed(42)
n = len(labels)
pos_scores = [scores[i] for i in range(n) if labels[i] == 1]
neg_scores = [scores[i] for i in range(n) if labels[i] == …How to Mock Shadow Mode Inference in Python
Simulates running multiple candidate models in shadow mode by adding randomized delays and returning their outputs alongside a primary model's output.
import random
import time
def shadow_mode_inference(candidates, mock_delay=0.1):
"""
Simulates running multiple candidate models in 'shadow mode'
by adding tiny randomized delays and returning their outputs
alongside the primary model's output.
"""
primary_output = "primary: answer"
shado…Browse by section
Each section groups closely related Python snippets.
ML engineering pipelines — Python code examples
What you will find here
This page collects ml engineering pipelines snippets — short, copy-ready Python you can paste into our free online IDE and run without installing anything. Each sample includes a plain-English explanation and the full source code.
Samples vs tutorials and challenges
Samples are quick reference — one concept per page. For step-by-step teaching, use our Python tutorials. To test yourself, try quizzes or coding challenges. Clean up style with the Python formatter.