Reference library

ML engineering pipelines

Feature prep, batch inference, model-serving hooks, and production ML workflow glue.

4 matches

How to Build a Mock TFX Pipeline in Python

Simulate a TFX-style ML pipeline with simple Python functions to understand component orchestration, data flow, and artifact passing.

tfx ml-pipeline orchestration

Python

# Mock TFX pipeline to illustrate component orchestration

def CsvExampleGen(data_path):
    """Mock component: Simulates reading CSV data."""
    print(f"ExampleGen: Reading from {data_path}")
    return {"records": 100, "name": "examples"}

def StatisticsGen(example_artifact):
    """Mock component: Simulates genera…

15 0 Open

ML engineering pipelines easy

How to Create a Mock Metaflow Flow in Python

Build a minimal Metaflow flow with two sequential steps that pass data between them using instance attributes.

metaflow ml-pipelines workflow

Python

from metaflow import FlowSpec, step, current


class MockFlow(FlowSpec):
    """A minimal Metaflow flow to demonstrate basic steps and branching."""

    @step
    def start(self):
        self.category = "mock"
        print(f"Start step for {self.category} flow")
        self.next(self.process)

    @step
    def pr…

15 0 Open

ML engineering pipelines easy

How to Define Dagster ML Assets in Python

Define a chain of Dagster software-defined assets that compute raw features, normalized features, and predictions for an ML pipeline.

dagster ml-pipeline asset

Python

from dagster import asset


@asset
def raw_features():
    return {"sepal_length": [5.1, 4.9, 6.2], "sepal_width": [3.5, 3.0, 3.4]}


@asset
def normalized_features(raw_features):
    values = raw_features["sepal_length"]
    mean = sum(values) / len(values)
    std = (sum((x - mean) ** 2 for x in values) / len(values…

13 0 Open

ML engineering pipelines easy

How to Simulate an Airflow ML Pipeline in Python

Mock an Airflow ML pipeline in plain Python by defining steps, simulating their execution with delays, and returning a success summary.

airflow ml pipeline

Python

from datetime import datetime, timedelta
import time


class MLPipeline:
    def __init__(self, pipeline_name):
        self.pipeline_name = pipeline_name
        self.steps = []

    def add_step(self, step_name, duration_seconds):
        self.steps.append({"name": step_name, "duration": duration_seconds})

    def …

13 0 Open

Browse by section

Each section groups closely related Python snippets.