Reference library

ML engineering pipelines

Feature prep, batch inference, model-serving hooks, and production ML workflow glue.

6 matches

How to Build a Data Validation Schema in Python

Create a lightweight validation schema using dataclasses and lambda validators to check fields in a dictionary.

import re
from dataclasses import dataclass, field
from typing import Any, Callable


@dataclass
class Field:
    name: str
    validator: Callable[[Any], bool]
    required: bool = True

    def validate(self, value: Any) -> bool:
        if not self.required and value is None:
            return True
        return …

12 0 Open

ML engineering pipelines easy

How to Create a Mock Metaflow Flow in Python

Build a minimal Metaflow flow with two sequential steps that pass data between them using instance attributes.

metaflow ml-pipelines workflow

Python

from metaflow import FlowSpec, step, current


class MockFlow(FlowSpec):
    """A minimal Metaflow flow to demonstrate basic steps and branching."""

    @step
    def start(self):
        self.category = "mock"
        print(f"Start step for {self.category} flow")
        self.next(self.process)

    @step
    def pr…

15 0 Open

ML engineering pipelines easy

How to Generate Experiment Tracking Run IDs in Python

Generate unique experiment run IDs with timestamps and random suffixes for tracking ML pipeline executions.

run-ids experiment-tracking ml-pipelines

Python

import random
import string
import time

def generate_run_id(prefix="exp"):
    timestamp = time.strftime("%Y%m%d_%H%M%S")
    suffix = "".join(random.choices(string.ascii_lowercase + string.digits, k=6))
    return f"{prefix}_{timestamp}_{suffix}"

if __name__ == "__main__":
    # Simulate tracking three experiment r…

13 0 Open

ML engineering pipelines easy

How to Load CSV Training Data in Python Without Pandas

Load CSV training data using Python's standard library and mock it with io.StringIO for testing, returning headers and rows as dictionaries.

csv ml-pipelines io-stringio

Python

import csv
from pathlib import Path


def load_csv_training_data(file_path: str | Path) -> tuple[list[str], list[dict[str, str]]]:
    """Load CSV training data and return headers plus rows as dictionaries."""
    with open(file_path, mode="r", newline="", encoding="utf-8") as csv_file:
        reader = csv.DictReader…

14 0 Open

ML engineering pipelines medium

How to Stage ML Model Workflows with Python Classes

Defines a Stage class to model ML pipeline stages with variants and mocks, printing grammar for Model, Staging, and Production stages.

ml-pipelines stages model-deployment

Python

class Stage:
    def __init__(self, name):
        self.name = name
        self.mocks = []
        self.variants = []

    def add_mock(self, mock_name):
        self.mocks.append(mock_name)

    def add_variant(self, variant_name, productions=()):
        self.variants.append((variant_name, list(productions)))

    …

12 0 Open

ML engineering pipelines medium

How to mock an artifact store with local paths in Python for ML pipelines

Create a temporary local artifact store with dummy files and metadata to test ML pipeline code without real storage.

ml-pipelines mock tempfile

Python

import tempfile
from pathlib import Path
import json


def create_artifact_store_mock(base_path: Path = None):
    """Create a local artifact store mock directory structure."""
    if base_path is None:
        base_path = Path(tempfile.mkdtemp())

    store_layout = {
        "artifacts": [
            {"name": "mode…

13 0 Open

Browse by section

Each section groups closely related Python snippets.