Reference library

ML engineering pipelines

Feature prep, batch inference, model-serving hooks, and production ML workflow glue.

6 matches

How to Build a Mock ML Pipeline with Prefect in Python

Create a lightweight Prefect flow with mock preprocessing, training, and evaluation tasks to prototype an ML pipeline end-to-end.

prefect machine-learning pipeline

Python

from prefect import task, flow
from datetime import datetime


@task
def preprocess_data(raw_value: float) -> float:
    """Mock preprocessing: normalize the input value."""
    return raw_value / 100.0


@task
def train_model(features: float) -> dict:
    """Mock training: return a fake model artifact."""
    return …

12 0 Open

ML engineering pipelines easy

How to Build a Mock TFX Pipeline in Python

Simulate a TFX-style ML pipeline with simple Python functions to understand component orchestration, data flow, and artifact passing.

tfx ml-pipeline orchestration

Python

# Mock TFX pipeline to illustrate component orchestration

def CsvExampleGen(data_path):
    """Mock component: Simulates reading CSV data."""
    print(f"ExampleGen: Reading from {data_path}")
    return {"records": 100, "name": "examples"}

def StatisticsGen(example_artifact):
    """Mock component: Simulates genera…

15 0 Open

ML engineering pipelines easy

How to Create a Mock Metaflow Flow in Python

Build a minimal Metaflow flow with two sequential steps that pass data between them using instance attributes.

metaflow ml-pipelines workflow

Python

from metaflow import FlowSpec, step, current


class MockFlow(FlowSpec):
    """A minimal Metaflow flow to demonstrate basic steps and branching."""

    @step
    def start(self):
        self.category = "mock"
        print(f"Start step for {self.category} flow")
        self.next(self.process)

    @step
    def pr…

15 0 Open

ML engineering pipelines easy

How to Define Dagster ML Assets in Python

Define a chain of Dagster software-defined assets that compute raw features, normalized features, and predictions for an ML pipeline.

dagster ml-pipeline asset

Python

from dagster import asset


@asset
def raw_features():
    return {"sepal_length": [5.1, 4.9, 6.2], "sepal_width": [3.5, 3.0, 3.4]}


@asset
def normalized_features(raw_features):
    values = raw_features["sepal_length"]
    mean = sum(values) / len(values)
    std = (sum((x - mean) ** 2 for x in values) / len(values…

13 0 Open

ML engineering pipelines easy

How to Simulate an Airflow ML Pipeline in Python

Mock an Airflow ML pipeline in plain Python by defining steps, simulating their execution with delays, and returning a success summary.

airflow ml pipeline

Python

from datetime import datetime, timedelta
import time


class MLPipeline:
    def __init__(self, pipeline_name):
        self.pipeline_name = pipeline_name
        self.steps = []

    def add_step(self, step_name, duration_seconds):
        self.steps.append({"name": step_name, "duration": duration_seconds})

    def …

13 0 Open

ML engineering pipelines medium

Training Pipeline Orchestration Mock DAG in Python

Build a mock DAG orchestrator that runs ML pipeline stages in dependency order using topological sorting (Kahn's algorithm).

dag pipeline topological-sort

Python

from collections import deque
from dataclasses import dataclass, field


@dataclass
class DAGNode:
    name: str
    task: callable
    dependencies: list[str] = field(default_factory=list)


class MockDAG:
    def __init__(self, nodes: list[DAGNode]):
        self.nodes = {n.name: n for n in nodes}
        self.execu…

13 0 Open

Browse by section

Each section groups closely related Python snippets.