Reference library

ML engineering pipelines

Feature prep, batch inference, model-serving hooks, and production ML workflow glue.

3 matches

How to Define Dagster ML Assets in Python

Define a chain of Dagster software-defined assets that compute raw features, normalized features, and predictions for an ML pipeline.

dagster ml-pipeline asset

Python

from dagster import asset


@asset
def raw_features():
    return {"sepal_length": [5.1, 4.9, 6.2], "sepal_width": [3.5, 3.0, 3.4]}


@asset
def normalized_features(raw_features):
    values = raw_features["sepal_length"]
    mean = sum(values) / len(values)
    std = (sum((x - mean) ** 2 for x in values) / len(values…

13 0 Open

ML engineering pipelines medium

How to Mock a Kubeflow Pipeline in Python

Build a minimal in-memory mock of a Kubeflow pipeline DAG using dataclasses and OrderedDict to chain component functions.

kubeflow pipelines mlops

Python

from typing import Dict, Any
from dataclasses import dataclass, field
from collections import OrderedDict


@dataclass
class KubeflowPipelineMock:
    """A minimal mock of a Kubeflow pipeline DAG."""
    name: str
    components: OrderedDict[str, callable] = field(default_factory=OrderedDict)

    def add_component(se…

14 0 Open

ML engineering pipelines medium

Training Pipeline Orchestration Mock DAG in Python

Build a mock DAG orchestrator that runs ML pipeline stages in dependency order using topological sorting (Kahn's algorithm).

dag pipeline topological-sort

Python

from collections import deque
from dataclasses import dataclass, field


@dataclass
class DAGNode:
    name: str
    task: callable
    dependencies: list[str] = field(default_factory=list)


class MockDAG:
    def __init__(self, nodes: list[DAGNode]):
        self.nodes = {n.name: n for n in nodes}
        self.execu…

13 0 Open

Browse by section

Each section groups closely related Python snippets.