Monitor Models with Logging

You've trained a model that performs brilliantly in your notebook — accuracy is high, the demo wows stakeholders, and you deploy it to production. Then, three weeks later, predictions silently drift, error rates climb, and users start complaining. The model didn't change; your data did — and you had no idea. This is the silent killer of AI applications, and the cure is structured, persistent logging. In this lesson, you'll learn how to monitor models with logging — turning raw predictions into an observability trail that reveals drift, performance degradation, and system health before they become incidents.

The problem this lesson solves

Model monitoring is not a "nice-to-have" — it's the difference between a reliable AI feature and a time bomb. Without logging, you're flying blind:

• Data drift changes the input distribution, so the model's assumptions break silently.
• Model degradation (concept drift) happens when the relationship between input and output shifts.
• Latency spikes and error bursts go unnoticed until users report them.
• Retraining decisions are guesswork because you have no historical context.

Logging is your only window into what your model actually does in production. It's not about storing every raw prediction — it's about capturing structured, queryable records that let you answer questions like "How did accuracy trend last week?" or "Which feature distribution changed most?"

Core concept / mental model

Think of model logging as a flight recorder for your AI system. Just as an airplane logs every sensor reading, your model should log every request, response, and relevant context. The recorder doesn't prevent problems — it gives you the data to diagnose and fix them.

In practice, monitoring models with logging means attaching a logging layer to your prediction pipeline that:

1. Captures the input, output, and metadata (timestamp, model version, feature hashes).
2. Stores it in a structured format (JSON lines, a database, or a log aggregator).
3. Surfaces it through dashboards or alerts so you can act.

The key distinction: application logging (debugging code) vs. model logging (understanding behavior). Model logging is a subset with AI-specific fields — think model_version, prediction_confidence, feature_importance, and drift_metrics.

How it works step by step

Let's break down the process of implementing model logging in your Python AI app.

Step 1: Decide what to log

Don't log everything — that's noise. Start with the essentials:

Step 2: Choose a structured format

JSON is the de facto standard. It's human-readable, parseable, and works with most logging tools. For high throughput, consider JSON Lines (each record on its own line) for append-only logs.

Step 3: Implement the logging layer

Wrap your prediction function with a decorator or a logging helper that inserts the metadata. In Python, use the standard logging module or a dedicated library like structlog for richer output.

Step 4: Handle failures and edge cases

Logging should never crash your app. Wrap logging calls in try/except, buffer async writes, and think about log retention policies.

Hands-on walkthrough

Let's implement a minimal but realistic model logger. We'll log predictions to a JSONL file, include drift signals, and then read them back for analysis.

Example 1: Basic structured logging with Python's logging

import json
import logging
from datetime import datetime, timezone
from functools import wraps

# Configure JSON formatter
class JsonFormatter(logging.Formatter):
    def format(self, record):
        log_entry = {
            "timestamp": datetime.now(timezone.utc).isoformat(),
            "level": record.levelname,
            "message": record.getMessage(),
            "model_version": getattr(record, "model_version", "unknown"),
            "latency_ms": getattr(record, "latency_ms", None),
        }
        return json.dumps(log_entry)

logger = logging.getLogger("model_monitor")
handler = logging.FileHandler("predictions.log")
handler.setFormatter(JsonFormatter())
logger.addHandler(handler)
logger.setLevel(logging.INFO)

def predict_with_logging(model, input_data):
    start = datetime.now(timezone.utc)
    prediction = model.predict(input_data)  # assume model has .predict
    latency_ms = (datetime.now(timezone.utc) - start).total_seconds() * 1000

    logger.info(
        "prediction_complete",
        extra={
            "model_version": "v1.2.0",
            "latency_ms": round(latency_ms, 2),
            "input_hash": hash(json.dumps(input_data, sort_keys=True)) % 100000,
        },
    )
    return prediction

# Simulate a model
class DummyModel:
    def predict(self, x):
        return sum(x)  # replace with real model logic

if __name__ == "__main__":
    model = DummyModel()
    result = predict_with_logging(model, [1, 2, 3])
    print("Prediction:", result)

Expected output (the actual hash will vary):

Prediction: 6

The log file predictions.log will contain lines like:

{"timestamp": "2025-04-02T10:15:30+00:00", "level": "INFO", "message": "prediction_complete", "model_version": "v1.2.0", "latency_ms": 0.12}

Example 2: Using structlog for richer context

structlog is a popular library that simplifies JSON logging. Install it with pip install structlog.

import structlog

log = structlog.get_logger()

def predict_and_log(model, input_data):
    log.msg("prediction_started", model_version="v1.2.0")
    try:
        prediction = model.predict(input_data)
        log.msg(
            "prediction_success",
            input_hash=hash(json.dumps(input_data)),
            prediction=str(prediction),
            model_version="v1.2.0",
            latency_ms=12.3,
        )
        return prediction
    except Exception as e:
        log.error("prediction_failed", error=str(e))
        raise

# Usage same as before, but output is already structured JSON to stdout

Example 3: Drift monitoring with logged data

The real power comes when you analyze the logs to detect drift. Here's a simple script that reads your JSONL log and computes the mean of a numeric feature over time.

import json
from datetime import datetime

def load_logs(path="predictions.log"):
    with open(path) as f:
        for line in f:
            try:
                yield json.loads(line)
            except json.JSONDecodeError:
                continue

def detect_drift(logs, feature_name="input_hash"):
    # Simple heuristic: compare recent versus baseline mean
    records = list(logs)
    baseline = [r.get(feature_name) for r in records[:100] if r.get(feature_name)]
    recent = [r.get(feature_name) for r in records[-100:] if r.get(feature_name)]
    if baseline and recent:
        return abs(sum(recent)/len(recent) - sum(baseline)/len(baseline))
    return 0.0

# In practice, you'd have a timestamp field and compute distributions.
# This is a placeholder to show the pattern.

This is a simplified example — real drift detection uses statistical tests (e.g., Kolmogorov–Smirnov) on feature distributions, but the logging layer is what makes it possible.

Compare options / when to choose what

You have several options for implementing model logging. Here's a comparison:

Choosing advice: Start with built-in logging for a prototype; move to structlog when you need consistent context; adopt a platform when you need team-wide visibility and automated alerts.

Troubleshooting & edge cases

Common pitfalls and how to handle them:

1. Logging slows down your prediction endpoint

• Symptom: Increased latency in production.
• Fix: Write logs asynchronously (e.g., queue.Queue + background thread) or use a batching library like python-json-logger with a buffered handler.

2. Too much log volume

• Symptom: Storage costs explode.
• Fix: Implement sampling — log every 10th request or log summaries (e.g., histograms) instead of raw data.

3. Sensitive data leakage

• Symptom: PII appears in logs (e.g., user emails).
• Fix: Hash or mask sensitive fields before logging. Use redaction libraries like loguru filters.

4. Logs don't have enough context for debugging

• Symptom: You see a prediction error but can't reproduce it.
• Fix: Include a request_id (UUID) in every log entry, and propagate it through your call stack.

5. JSON serialization fails on non-serializable objects

• Symptom: TypeError: Object of type ndarray is not JSON serializable
• Fix: Use default=str in your json.dumps, or convert numpy arrays to lists before logging.

What you learned & what's next

You've just built a foundation for monitoring models with logging — you understand why logging is critical for catching drift and degradation, how to structure logs with JSON, and how to compare tools like structlog vs. built-in logging. You also practiced writing log entries and reading them back for analysis.

Next lesson in the Applied AI path will take this further — likely covering automated drift detection or dashboarding to visualize your logs in real time. Regardless, your logging layer is now in place to support those advanced observability features.

Remember: a model you can't observe is a model you can't trust.