Set up retraining triggers

Your model was a hero at launch — precision high, latency low, users happy. Then, quietly, the world shifted. New user patterns emerged, the data distribution drifted, and your carefully tuned model started making decisions on stale ground truth. The pain is real: models don't degrade on a schedule, and waiting for a quarterly retraining cycle means your AI is confidently wrong for weeks. Setting up model retraining triggers is how you stop flying blind and start responding to drift the moment it matters.

The problem this lesson solves

Machine learning models are not fire-and-forget artifacts. The moment you deploy, the world starts moving: new user segments appear, seasonal trends kick in, or your own product changes how data is collected. The result is concept drift (the relationship between input and output changes) and data drift (the distribution of your input features shifts). When either happens, your model's performance decays silently.

The core problem? Performance decay is invisible until it hurts. You won't see the accuracy drop in production logs because you're not computing ground truth labels in real time. Without triggers, you retrain on a fixed schedule — monthly, quarterly — which is either too slow (you lose money during drift periods) or too wasteful (you retrain when nothing changed). The solution is a trigger system: a mechanism that watches signals, detects meaningful change, and kicks off a retraining pipeline automatically.

By the end of this lesson, you'll be able to set up model retraining triggers that are both responsive and cost-aware — a core skill in Applied AI engineering.

Core concept / mental model

Think of a smoke detector in your house. It doesn't check the air quality every hour on a schedule; it continuously monitors for a specific signal (smoke particles) and triggers an alarm when that signal crosses a threshold. Likewise, a retraining trigger is a monitoring loop that watches key indicators and starts a retraining job when a threshold is breached.

The mental model has three layers:

1. Signal: What you monitor — e.g., prediction distribution, feature distribution, live performance metrics, or business KPIs.
2. Threshold: The level of change that warrants action — e.g., "accuracy dropped by 5%" or "PSI > 0.2".
3. Action: The retraining pipeline — data extraction, training, evaluation, and deployment.

Here's a simple visual (in words):

[Live data stream] --> [Monitor: compute signal] --> [Compare to threshold] --> [Trigger retraining?] --> [Launch pipeline]

This is often called a reactive trigger (as opposed to a scheduled trigger). Reactive triggers are the gold standard for MLOps because they react to change, not time.

How it works step by step

Setting up a retraining trigger involves six logical steps:

1. Define what to monitor: Choose the most informative signals. Common choices: - Prediction drift: How the distribution of your model's outputs changes (e.g., % of positive predictions). - Data drift: Comparing feature distributions between training and live data (often using PSI or KS test). - Live performance: Only possible if you have delayed ground truth (e.g., churn after 30 days). - Business metrics: e.g., conversion rate or revenue per user.

2. Set up a monitoring job: A scheduled or streaming process that computes the signal at intervals (daily, hourly). This is typically a small Python job that pulls recent data and computes metrics.

3. Define thresholds: Choose thresholds that balance false alarms (too many retrains, high cost) and missed drift (too few). Thresholds are often based on historical variance or domain expertise.

4. Implement the trigger logic: A simple if statement in the monitoring job that calls the retraining pipeline when the threshold is crossed.

5. Integrate with retraining pipeline: Your trigger should call the same pipeline you use for manual retraining — reusing code keeps things consistent.

6. Log and alert: Always log trigger events. If a retrain is triggered, you want to know why (drift or performance) to audit decisions.

Pro tip: Start with a simple metric like prediction drift — it's cheap to compute (no ground truth needed) and often correlates with data drift.

Hands-on walkthrough

Let's build a minimal but complete retraining trigger system in Python. We'll use a mock monitoring job that checks for prediction drift and triggers a retraining script.

Step 1: Compute the signal

We'll track the fraction of positive predictions in a rolling window, compare it to the training-time baseline.

# monitor.py
import numpy as np
from datetime import datetime, timedelta

TRAIN_BASELINE_POSITIVE_RATE = 0.35  # from training data
WINDOW_DAYS = 7


def compute_positive_rate(predictions):
    """Given list of 0/1 predictions, return fraction of positives."""
    return np.mean(predictions) if len(predictions) > 0 else 0.0


def get_recent_predictions():
    # Simulate pulling from a database or streaming service
    # In production: query your prediction log table
    return np.random.binomial(1, 0.5, 1000)  # placeholder


# One monitoring run
def check_and_trigger():
    recent_preds = get_recent_predictions()
    live_rate = compute_positive_rate(recent_preds)
    print(f"Live positive rate: {live_rate:.3f}, baseline: {TRAIN_BASELINE_POSITIVE_RATE}")

    # Trigger if abs diff > 0.1 (10 percentage points)
    if abs(live_rate - TRAIN_BASELINE_POSITIVE_RATE) > 0.1:
        print("Trigger retraining!")
        # In real system: call retraining pipeline
    else:
        print("No retraining needed")

if __name__ == "__main__":
    check_and_trigger()

Expected output (varies):

Live positive rate: 0.487, baseline: 0.35
Trigger retraining!

Step 2: Add data drift detection with PSI

For feature drift, compute the Population Stability Index between training and live feature distributions for key features.

# psi.py
import numpy as np

def psi(expected, actual, buckets=10):
    """Calculate PSI between two arrays."""
    expected_hist, bin_edges = np.histogram(expected, bins=buckets)
    actual_hist, _ = np.histogram(actual, bins=bin_edges)

    # Avoid zeros
    expected_hist = np.where(expected_hist == 0, 1e-6, expected_hist)
    actual_hist = np.where(actual_hist == 0, 1e-6, actual_hist)

    expected_pct = expected_hist / expected_hist.sum()
    actual_pct = actual_hist / actual_hist.sum()

    psi_val = np.sum((actual_pct - expected_pct) * np.log(actual_pct / expected_pct))
    return psi_val

# Example
live_feature = np.random.normal(0.1, 1.0, 10000)
train_feature = np.random.normal(0.0, 1.0, 10000)
print(f"PSI: {psi(train_feature, live_feature):.3f}")

Output: PSI: 0.012 (low drift). If PSI > 0.1, you may want to retrain.

Step 3: Wire it together with a scheduler

Use cron or APScheduler to run the check daily. Here's a simple example using APScheduler:

# scheduler.py
from apscheduler.schedulers.blocking import BlockingScheduler
import monitor

sched = BlockingScheduler()

@sched.scheduled_job('cron', hour=2)  # run at 2 AM daily

def scheduled_check():
    monitor.check_and_trigger()

print("Monitoring started")
sched.start()

Step 4: Retraining pipeline stub

Your trigger should call a function that runs the full pipeline — here's a stub:

# retrain.py
def retrain_model(trigger_reason):
    print(f"Retraining due to: {trigger_reason}")
    # In real world: extract fresh data, train model, evaluate, deploy
    # e.g., model_url = train_and_deploy()
    return "new_model_v2"

Put it all together: when check_and_trigger detects drift, it calls retrain_model. You can extend with alerting (e.g., send email or Slack message).

Compare options / when to choose what

There are three main trigger strategies you'll encounter in MLOps:

Variation: Performance-based triggers — if you have delayed labels (e.g., user churn known after 30 days), you can compute live accuracy periodically and trigger on decline. This requires a delayed labeling pipeline and is more accurate for business impact.

Variation: Statistical tests — instead of a simple threshold, use KS test or culmulative sum (CUSUM) to detect drift with statistical significance. This reduces false alarms when data is noisy.

Variation: Business KPI triggers — monitor business metrics (e.g., weekly active users) and trigger if they drop. This is the most direct link to revenue, but is a lagging indicator.

Troubleshooting & edge cases

Edge case 1: Seasonal data

Retail models see huge drift around holidays. A reactive trigger will fire too often, causing excessive retrains. Fix: Set thresholds relative to seasonal baselines, or use hybrid (retrain at holidays anyway).

Edge case 2: Noisy signals

If your live data is small (e.g., low volume), short-window metrics are noisy. Fix: Use a longer rolling window or statistical tests to reduce false alarms.

Edge case 3: Threshold not tuned

A threshold too low causes frequent retrains; too high misses drift. Fix: Start with a threshold based on historical variance (e.g., mean + 2 standard deviations) and adjust after monitoring false positives.

Troubleshooting: "My trigger never fires"

• Check your baseline was computed on the correct training set — a biased baseline will make the trigger useless.
• Verify your live data query is actually pulling recent data (timezone issues are a classic).
• Log the computed signal values — you may be looking at a normalized version that hides drift.

Troubleshooting: "Model gets worse after retraining"

• The new data might have label noise or a distribution shift that the model can't learn. Use a holdout set to validate before deploying.
• Ensure your retraining pipeline preprocesses data identically to the original.
• Consider a canary deployment: serve 10% of traffic to the new model and compare.

What you learned & what's next

You now understand the core idea behind setting up model retraining triggers: moving from time-based to change-based retraining. You can distinguish between data drift and concept drift, compute simple signals like positive-rate drift and PSI, and implement a trigger that calls a retraining pipeline. You've also learned how to choose between scheduled, reactive, and hybrid strategies based on domain needs.

Key takeaways:

• Retraining triggers are essential for production ML to maintain accuracy in changing environments.
• The mental model is signal → threshold → action.
• Start simple: monitor prediction drift first, then add feature drift with PSI.
• Use statistical tests to reduce false alarms.
• Always log trigger events and validate new models before full deployment.

Next step: The next lesson in the Applied AI engineering track covers model monitoring dashboards — you'll learn how to visualize these signals and alert your team when retraining happens. That will close the loop on a robust production ML workflow.