From Notebook to Production: Reproducible Python Environments with Docker and Poetry

Here is the article, written as requested for PythonSkillset.com.

From Notebook to Production: A Practical Guide to Reproducible Python Environments with Docker and Poetry

You have a beautiful Jupyter notebook. The code runs perfectly on your machine. You share it with a colleague, and suddenly nothing works. The dreaded "it works on my machine" problem has struck again.

We have all been there. Python is fantastic for rapid prototyping, but its flexibility becomes a curse when we need to move code from a local experiment to a shared server or a production system. The solution is to lock down everything: the Python version, the libraries, the operating system libraries.

Two tools have become the standard for solving this: Docker and Poetry. Docker handles the entire system, while Poetry manages your Python dependencies with precision. Together, they create a truly portable environment.

This guide walks through a practical workflow to go from a messy notebook to a clean, reproducible environment that you can deploy with confidence.

Why Not Just Use requirements.txt?

Many projects use pip freeze > requirements.txt. This works only if you and your target system have the exact same Python version and operating system. It does not capture the system-level dependencies like C compilers that many Python packages (like numpy or pandas) need.

Docker creates a "fake" computer with a known operating system (Ubuntu, for example). Poetry, on the other hand, creates a poetry.lock file that pins every single dependency to an exact version hash. This combination is the gold standard for reproducibility.

Step 1: Setting Up Your Project with Poetry

First, you need to organize your notebook-based code into a proper package. Let us assume you have a script called train_model.py that uses pandas, scikit-learn, and xgboost.

If you do not have Poetry installed, you can get it from their official website.

# Create a new project structure
poetry new my-ml-workflow
cd my-ml-workflow

This creates a folder with a standard layout:

my-ml-workflow/
├── pyproject.toml
├── README.md
├── my_ml_workflow/
│   └── __init__.py
└── tests/
    └── __init__.py

Now, add your initial dependencies.

poetry add pandas scikit-learn xgboost

Poetry will automatically create a poetry.lock file. This file is critical. You must commit it to your version control system (like Git). It locks the version of every installed package and its dependencies.

You can also specify the exact Python version you need in pyproject.toml:

[tool.poetry.dependencies]
python = "3.10"

Step 2: Containerizing with Docker

Now we create a Dockerfile that uses Poetry to install the environment inside a lightweight container.

Here is a good starting Dockerfile:

# Use a slim Python image as a base
FROM python:3.10-slim

# Set the working directory
WORKDIR /app

# Install Poetry (official installer)
RUN pip install poetry

# Copy only the lock files first to leverage Docker caching
COPY pyproject.toml poetry.lock ./

# Configure poetry to not create a virtual environment inside the container
# and install the project dependencies
RUN poetry config virtualenvs.create false \
    && poetry install --no-interaction --no-ansi

# Copy the rest of the application code
COPY . .

# Specify the command to run your script
CMD ["python", "train_model.py"]

Step 3: Building and Running Your Container

Building the image is straightforward:

docker build -t my-ml-workflow:latest .

To run it:

docker run --rm my-ml-workflow:latest

If your script requires data, you will need to mount a volume. For example:

docker run --rm -v $(pwd)/data:/app/data my-ml-workflow:latest

Step 4: Handling Common Pitfalls

Problem 1: GPU Support for Machine Learning If you need CUDA support, you need to change the base image. Use nvidia/cuda or the official tensorflow/tensorflow:latest-gpu images as a starting point, then add Poetry on top. This is more complex but essential for deep learning.

Problem 2: System Libraries (e.g., for psycopg2 or opencv) Some Python packages need system-level libraries. You install them in the Dockerfile before Poetry. For example:

RUN apt-get update && apt-get install -y \
    gcc \
    libpq-dev \
    && rm -rf /var/lib/apt/lists/*

Problem 3: The Notebook Itself You can run a Jupyter notebook inside a Docker container. Use jupyter/datascience-notebook as a base or install Jupyter via Poetry. Then expose port 8888. This lets your colleague start the exact same notebook with the exact same libraries.

Real-World Example: A PythonSkillset Data Pipeline

At PythonSkillset, we had a project where an analyst built a complex feature engineering notebook using pandas and numpy. The notebook worked perfectly on their Windows machine with Python 3.9. When they tried to move it to our Linux server running Python 3.11, the numpy binary was incompatible.

The fix was simple in retrospect. The analyst used Poetry to create a lock file, and we built a Docker image from that lock file. The project went from "error on every platform" to running identically on development laptops, staging servers, and production.

Conclusion

The combination of Docker and Poetry is not just for software engineers working on web apps. It is equally critical for data scientists, machine learning engineers, and anyone who needs to share Python code that works. It stops the insanity of debugging environment issues.

The next time you finish a notebook and think, "I need to put this somewhere," take the extra hour to set up a poetry project and a dockerfile. It will save you countless hours of frustration later. Your future self and your colleagues will thank you.