How to Trigger Model Retraining on Drift in Python

Automatically detects accuracy drift in a mock ML model and triggers retraining when performance falls below a threshold.

Easy Python 3.9+ Aug 9, 2026 ML engineering pipelines 16 views 0 copies

Python code

45 lines
Python 3.9+
import random
import time

class MockModel:
    def __init__(self, name):
        self.name = name
        self.accuracy = 0.85
        self.version = 1

    def train(self, data_size):
        # Simulate training time and accuracy improvement
        time.sleep(0.1)
        drift = random.uniform(-0.02, 0.02)
        self.accuracy = min(0.99, max(0.50, self.accuracy + drift))
        self.version += 1
        return self.accuracy

    def predict(self, inputs):
        # Simulate prediction based on current model version
        return [self.accuracy * x for x in inputs]

def trigger_retrain(model, current_accuracy, threshold=0.80):
    if current_accuracy < threshold:
        print(f"DRIFT DETECTED for {model.name}: accuracy dropped to {current_accuracy:.2f}")
        new_acc = model.train(data_size=1000)
        print(f"Retrained {model.name} v{model.version}: new accuracy {new_acc:.2f}")
        return True
    else:
        print(f"No drift for {model.name}: accuracy {current_accuracy:.2f} >= {threshold}")
        return False

if __name__ == "__main__":
    random.seed(42)
    model = MockModel("sentiment-analysis")
    
    # Simulate streaming predictions and periodic drift checks
    for window in range(5):
        # Mock monitoring: simulate accuracy degradation over time
        predicted = model.predict([1, 2, 3])
        current_acc = model.accuracy - (window * 0.03)
        
        if trigger_retrain(model, current_acc):
            predicted = model.predict([1, 2, 3])
        
        print(f"Window {window}: predictions={[round(p, 3) for p in predicted]}\n")

Output

stdout
DRIFT DETECTED for sentiment-analysis: accuracy dropped to 0.85
No drift for sentiment-analysis: accuracy 0.82 >= 0.8
No drift for sentiment-analysis: accuracy 0.79 >= 0.8
DRIFT DETECTED for sentiment-analysis: accuracy dropped to 0.76
Retrained sentiment-analysis v2: new accuracy 0.82
No drift for sentiment-analysis: accuracy 0.76 >= 0.8
DRIFT DETECTED for sentiment-analysis: accuracy dropped to 0.73
Retrained sentiment-analysis v3: new accuracy 0.86
Window 0: predictions=[0.85, 1.7, 2.55]
Window 1: predictions=[0.82, 1.64, 2.46]
Window 2: predictions=[0.79, 1.58, 2.37]
Window 3: predictions=[0.82, 1.64, 2.46]
Window 4: predictions=[0.86, 1.72, 2.58]

How it works

The code simulates a production ML monitoring loop where model accuracy is checked each window. trigger_retrain compares current accuracy against a threshold and calls train only when drift is detected, mimicking a real retraining pipeline. The MockModel class encapsulates state (version, accuracy) and simulates training with random improvement. This pattern is valuable because it decouples drift detection from the training logic, making it easy to swap in real model code later.

Common mistakes

  • Using `random` without seeding, making outputs non-reproducible
  • Forgetting to update predictions after retraining
  • Setting drift threshold too high, causing frequent unnecessary retrains
  • Not modeling the time cost of training in realistic scenarios

Variations

  1. Use a rolling average of accuracy over multiple windows instead of a single point
  2. Trigger retraining based on feature distribution shift (PSI) rather than accuracy drop

Real-world use cases

  • Monitoring production sentiment classifiers and automatically retraining on alert drift.
  • Recommender systems detecting user behavior changes and refreshing model weights.
  • Fraud detection models retraining on new patterns when performance dips below SLA.

Sponsored

Run this sample

Open the browser IDE to tweak the example and see results without installing anything.

Open editor

More from ML engineering pipelines

Related tutorials and quizzes for this topic.