Convert Notebooks to Production Code

Your Jupyter notebook is a miracle of exploration: it got you to a working model, a clean accuracy score, and a satisfying final plot. But now the real work begins. That notebook — with its 47 hidden states, magic commands, and one cell that absolutely must run before the cell above it — is a liability the moment it touches a production environment. It's slow to retrain, impossible to unit test, and a nightmare for your team to review. The fix is a disciplined conversion from notebook to production code: a process that turns your experimental scribbles into modular, tested, maintainable Python packages. This lesson gives you the exact step-by-step playbook to do it without losing your mind — or your model's performance.

The Problem This Lesson Solves

Jupyter notebooks are the perfect environment for exploration — they let you iterate visually, inspect dataframes cell by cell, and tweak parameters with instant feedback. But they are the worst environment for production. Here's the pain:

• Hidden state: Cell execution order is not the file's line order. Variables persist across cells, and if you rerun the notebook out of order (or restart and run all), you get different results. Production code must be deterministic.
• No structure: Functions are scattered, magic commands (%matplotlib inline) do nothing outside a notebook kernel, and global variables create invisible coupling.
• No testability: You can't pytest a notebook cell. Verification is manual — you look at the output and hope it's right.
• No versioning: Notebook diffs are JSON blob nightmares. Code review of a notebook is basically impossible.
• Performance & scaling: Notebooks keep data in memory, don't offer easy parallelization, and lack the entry points (CLI, API) that production systems need.

If you've ever tried to wrap a notebook in a Flask app or run it on a schedule, you've hit this wall. This lesson is the sledgehammer that breaks it down.

Core Concept / Mental Model

Think of your notebook as a prototype car — duct-taped, battery exposed, with a note taped to the steering wheel: "Don't turn left too fast." It drives, but it's not safe for the highway. Converting it to production code is like turning that prototype into a manufactured vehicle — with an engine, a chassis, and a manual. The process is not rewriting from scratch; it's a structured refactoring that preserves the functional core (the model, the logic) while making it robust and deployable.

At the heart is the separation of concerns. Your notebook mixes five distinct responsibilities into one soup:

1. Data loading — reading raw files, connecting to databases
2. Preprocessing — cleaning, feature engineering
3. Model training — fitting and evaluating
4. Inference — making predictions on new data
5. Reporting — visualizations, metrics, output artifacts

Production code splits these into modules (files), each with a single purpose. The notebook becomes a driver script — a thin layer that orchestrates the modules, often from the command line. Another useful analogy: the notebook is your recipe on a sticky note; production code is the cookbook page — standardized, ingredient list separate from instructions, tested by others.

Key terms you'll need:

• Module: A .py file that groups related functions/classes.
• Package: A directory of modules with an __init__.py.
• Entry point: A function the runner calls (e.g., main() for a CLI, predict() for an API).
• Configuration: Parameters (paths, hyperparameters) kept outside code — in YAML, env vars, or dataclasses.

How It Works Step by Step

Follow this battle-tested sequence. It works for both an 8-cell notebook and a 40-cell monster.

Step 1: Inventory the Cells

Read your notebook top to bottom and group cells by their function: loading, cleaning, feature, train, evaluate, plot. Create a quick map — either in your head or in a comment at the top of a new script.

Step 2: Extract into Functions

Turn every step's cells into one or more pure functions — functions that take inputs, return outputs, and have no side effects. This is the single most important change. Pure functions are testable, cacheable, and parallelizable.

# Before (notebook cell)
# df = pd.read_csv('data.csv')
# df['total'] = df['price'] * df['quantity']
# df['category'] = df['total'].apply(lambda x: 'high' if x > 100 else 'low')

# After (production module)
def load_data(path: str) -> pd.DataFrame:
    return pd.read_csv(path)

def add_total_and_category(df: pd.DataFrame) -> pd.DataFrame:
    df = df.copy()  # avoid SettingWithCopyWarning
    df['total'] = df['price'] * df['quantity']
    df['category'] = np.where(df['total'] > 100, 'high', 'low')
    return df

Step 3: Remove Global State and Magic

Replace % magics with standard Python. Move global variables (like DATA_PATH = 'data.csv') into a config.py or a @dataclass.

Step 4: Create Modules and a Package

Group functions into files: data_loader.py, features.py, model.py, evaluate.py. Add an empty __init__.py to make it a package.

Step 5: Write a Driver Script

Create a main.py that calls the modules in the correct order, with a main() function and if __name__ == '__main__':. This is your notebook's replacement — run it with python main.py and get the same results.

Step 6: Add Tests

Write unit tests for the pure functions using pytest. Test the edge cases your notebook ignored.

Step 7: Package and Document

Add a pyproject.toml (or setup.py) and a README.md. Now it's installable and shareable.

Hands-On Walkthrough

Let's convert a small but realistic notebook that builds a simple classifier. We'll go cell by cell.

The Original Notebook (Mental Snapshot)

• Cell 1: import pandas as pd; import matplotlib.pyplot as plt
• Cell 2: df = pd.read_csv('data/iris.csv')
• Cell 3: X = df.drop('species', axis=1); y = df['species']
• Cell 4: from sklearn.model_selection import train_test_split; X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2)
• Cell 5: from sklearn.ensemble import RandomForestClassifier; clf = RandomForestClassifier(n_estimators=100); clf.fit(X_tr, y_tr)
• Cell 6: print(clf.score(X_te, y_te))
• Cell 7: plt.scatter(df['sepal_length'], df['sepal_width'], c=y.astype('category').cat.codes)

Step 1 & 2: Create the package structure

my_model/
  __init__.py
  data.py
  features.py
  model.py
  evaluate.py
  main.py
  tests/
    test_features.py

Step 3: Write the modules

# data.py
import pandas as pd

def load_data(path: str) -> pd.DataFrame:
    return pd.read_csv(path)


# features.py
from typing import Tuple
import pandas as pd
from sklearn.model_selection import train_test_split

def extract_features_and_target(df: pd.DataFrame, target_col: str) -> Tuple[pd.DataFrame, pd.Series]:
    X = df.drop(target_col, axis=1)
    y = df[target_col]
    return X, y

def split_data(X: pd.DataFrame, y: pd.Series) -> Tuple[pd.DataFrame, pd.DataFrame, pd.Series, pd.Series]:
    return train_test_split(X, y, test_size=0.2, random_state=42)


# model.py
from sklearn.ensemble import RandomForestClassifier
from sklearn.base import BaseEstimator

def train_model(X_tr: pd.DataFrame, y_tr: pd.Series) -> BaseEstimator:
    clf = RandomForestClassifier(n_estimators=100, random_state=42)
    clf.fit(X_tr, y_tr)
    return clf

def predict(model: BaseEstimator, X: pd.DataFrame):
    return model.predict(X)


# evaluate.py
from typing import Any
import pandas as pd
from sklearn.metrics import accuracy_score

def evaluate(model: Any, X_te: pd.DataFrame, y_te: pd.Series) -> float:
    y_pred = model.predict(X_te)
    return accuracy_score(y_te, y_pred)

Step 5: Write the driver script

# main.py
from my_model.data import load_data
from my_model.features import extract_features_and_target, split_data
from my_model.model import train_model
from my_model.evaluate import evaluate


def main():
    # 1. Load
    df = load_data('data/iris.csv')

    # 2. Preprocess
    X, y = extract_features_and_target(df, target_col='species')
    X_tr, X_te, y_tr, y_te = split_data(X, y)

    # 3. Train
    model = train_model(X_tr, y_tr)

    # 4. Evaluate
    acc = evaluate(model, X_te, y_te)
    print(f'Test accuracy: {acc:.3f}')


if __name__ == '__main__':
    main()

Run it:

$ python main.py
Test accuracy: 1.000

Same result as your notebook cell — but now every function is reusable and testable.

Step 6: Add a test

# tests/test_features.py
import pandas as pd
from my_model.features import extract_features_and_target

def test_extract_features_and_target():
    df = pd.DataFrame({'a': [1, 2], 'b': [3, 4], 'target': ['x', 'y']})
    X, y = extract_features_and_target(df, 'target')
    assert list(X.columns) == ['a', 'b']
    assert y.tolist() == ['x', 'y']

Run pytest — you've just taken your first step into production-grade verification.

Compare Options / When to Choose What

You don't have to manually refactor. Here are the common alternatives:

When to choose what

• If you need a prototype to run in a day: nbconvert or papermill.
• If you're building a service (Flask/FastAPI) or package: manual refactoring.
• If your team can't live without notebooks: nbdev.

Pro tip: Start with manual refactoring for your first conversion. It teaches you the structure patterns you'll reuse everywhere. Tools like nbdev are powerful but hide those patterns.

Troubleshooting & Edge Cases

My output differs after conversion — what's wrong?

• Random seeds: Your notebook didn't set random_state; adding one to train_test_split and the model makes it reproducible. If you didn't set it previously, results will differ. That's expected — you're better off.
• Data order: train_test_split defaults to a random split; your notebook may have used shuffle=False or a different seed. Match them to get identical splits.
• In-place mutations: A function that modifies the input DataFrame (e.g., df['x'] = ...) can sneak into your module and cause unexpected changes if you forgot .copy(). Always copy inside functions.

I get NameError: name 'X' is not defined in my script

• You likely moved code into functions but forgot to pass arguments. In notebooks, globals are implicit; in functions, they're explicit. Check your function signatures.

matplotlib inline magic breaks

• Use plt.savefig() instead of plt.show() in production scripts, and call plt.close() to free memory.

My tests fail on Windows because of file paths

• Use pathlib.Path or os.path.join in your code, not hard-coded / separators. Better: put paths in a config file.

Dependency hell (missing packages in production)

• Use a virtual environment and an environment.yml or requirements.txt. Export it immediately after your first successful run.

What You Learned & What's Next

You can now convert notebooks to production code — a transformation that makes your models not just a result but a reliable asset. Specifically, you can:

• Explain the core idea: turn an experimental notebook into a modular, tested package by separating data loading, preprocessing, training, inference, and evaluation.
• Complete a practical exercise: refactor a real notebook into main.py plus modules, add tests, and run it from the command line.

Your new mental model — likely to stick: notebook is exploration, production is engineering. You've learned to distil the essential logic into pure functions and wrap them with structure.

Next in the Applied AI engineering track, you'll learn how to deploy this production code — turning your main.py into a REST API or a scheduled job. That's where the clean separation you just built pays off: each module can be scaled independently, and your service is testable from day one.

Go convert that notebook — your future self (and your team) will thank you.