How to Mock Kedro Pipeline Nodes in Python

Create a modular Kedro pipeline with node functions, namespacing, and input/output mapping to mock pipeline execution locally.

Medium Python 3.9+ Aug 9, 2026 ML engineering pipelines 15 views 0 copies

kedro pipeline modular mlops machine-learning data-engineering
Requires third-party packages — install first
pip install kedro

Python code

41 lines
Python 3.9+ 
from kedro.pipeline import Pipeline, node
from kedro.pipeline.modular_pipeline import pipeline as modular_pipeline


def preprocess(data: list) -> list:
    """Clean data by removing None values."""
    return [item for item in data if item is not None]


def transform(data: list) -> list:
    """Add 1 to each numeric value."""
    return [x + 1 for x in data]


def export(data: list) -> dict:
    """Convert to dictionary with counts."""
    return {"values": data, "count": len(data)}


if __name__ == "__main__":
    raw_data = [1, 2, None, 4, 5]

    preprocess_node = node(preprocess, inputs="raw", outputs="clean", name="preprocess")
    transform_node = node(transform, inputs="clean", outputs="transformed", name="transform")
    export_node = node(export, inputs="transformed", outputs="result", name="export")

    pipeline = modular_pipeline(
        Pipeline([preprocess_node, transform_node, export_node]),
        namespace="data_pipeline",
        inputs={"raw": "data_input"},
        outputs={"result": "final_output"},
    )

    print(f"Pipeline nodes: {[node.name for node in pipeline.nodes]}")
    print(f"Pipeline inputs: {pipeline.inputs()}")
    print(f"Pipeline outputs: {pipeline.outputs()}")

    clean = preprocess(raw_data)
    transformed = transform(clean)
    result = export(transformed)
    print(f"Execution result: {result}")

Output

stdout 
Pipeline nodes: ['data_pipeline.preprocess', 'data_pipeline.transform', 'data_pipeline.export']
Pipeline inputs: {'data_input'}
Pipeline outputs: {'final_output'}
Execution result: {'values': [2, 3, 5, 6], 'count': 4}

How it works

Kedro's node function wraps Python functions with explicit input/output names, while modular_pipeline groups related nodes under a namespace. The namespace parameter prefixes node names for organized logs and tracing, while inputs and outputs map external dataset names to internal ones. When run locally, only the __main__ block executes the functions directly, allowing full pipeline logic to be mocked and verified without a full Kedro run.

Common mistakes

  • Forgetting that `node` input/output names must match dataset names in the catalog before running via `kedro run`
  • Using the same dataset name for both an input and output without an intermediate transformation
  • Misunderstanding `namespace` — it prefixes node names but not dataset names unless explicitly mapped
  • Assuming `modular_pipeline` requires a namespace; it's optional for simple pipelines

Variations

  1. Use `Pipeline([...])` alone without modular wrapping when you need no namespacing or external mapping
  2. Create pipeline objects dynamically from lists of node tuples for cleaner configuration

Real-world use cases

  • Prototyping an ML feature-engineering workflow locally before wiring data catalog dependencies.
  • Unit-testing pipeline node logic in CI by mocking external datasets with simple in-memory lists.
  • Documenting data transformations for team reviews using explicit input/output dataset mappings.

Sponsored

Run locally

This sample needs third-party packages, so it cannot run in the browser IDE. Copy the code above, install the packages shown at the top, then run it in your own Python environment.

More from ML engineering pipelines

Keep learning

Related tutorials and quizzes for this topic.

Tutorials

Quizzes