How to Trigger Model Retraining on Drift in Python
Automatically detects accuracy drift in a mock ML model and triggers retraining when performance falls below a threshold.
Python code
45 linesimport random
import time
class MockModel:
def __init__(self, name):
self.name = name
self.accuracy = 0.85
self.version = 1
def train(self, data_size):
# Simulate training time and accuracy improvement
time.sleep(0.1)
drift = random.uniform(-0.02, 0.02)
self.accuracy = min(0.99, max(0.50, self.accuracy + drift))
self.version += 1
return self.accuracy
def predict(self, inputs):
# Simulate prediction based on current model version
return [self.accuracy * x for x in inputs]
def trigger_retrain(model, current_accuracy, threshold=0.80):
if current_accuracy < threshold:
print(f"DRIFT DETECTED for {model.name}: accuracy dropped to {current_accuracy:.2f}")
new_acc = model.train(data_size=1000)
print(f"Retrained {model.name} v{model.version}: new accuracy {new_acc:.2f}")
return True
else:
print(f"No drift for {model.name}: accuracy {current_accuracy:.2f} >= {threshold}")
return False
if __name__ == "__main__":
random.seed(42)
model = MockModel("sentiment-analysis")
# Simulate streaming predictions and periodic drift checks
for window in range(5):
# Mock monitoring: simulate accuracy degradation over time
predicted = model.predict([1, 2, 3])
current_acc = model.accuracy - (window * 0.03)
if trigger_retrain(model, current_acc):
predicted = model.predict([1, 2, 3])
print(f"Window {window}: predictions={[round(p, 3) for p in predicted]}\n")
Output
DRIFT DETECTED for sentiment-analysis: accuracy dropped to 0.85
No drift for sentiment-analysis: accuracy 0.82 >= 0.8
No drift for sentiment-analysis: accuracy 0.79 >= 0.8
DRIFT DETECTED for sentiment-analysis: accuracy dropped to 0.76
Retrained sentiment-analysis v2: new accuracy 0.82
No drift for sentiment-analysis: accuracy 0.76 >= 0.8
DRIFT DETECTED for sentiment-analysis: accuracy dropped to 0.73
Retrained sentiment-analysis v3: new accuracy 0.86
Window 0: predictions=[0.85, 1.7, 2.55]
Window 1: predictions=[0.82, 1.64, 2.46]
Window 2: predictions=[0.79, 1.58, 2.37]
Window 3: predictions=[0.82, 1.64, 2.46]
Window 4: predictions=[0.86, 1.72, 2.58]
How it works
The code simulates a production ML monitoring loop where model accuracy is checked each window. trigger_retrain compares current accuracy against a threshold and calls train only when drift is detected, mimicking a real retraining pipeline. The MockModel class encapsulates state (version, accuracy) and simulates training with random improvement. This pattern is valuable because it decouples drift detection from the training logic, making it easy to swap in real model code later.
Common mistakes
- Using `random` without seeding, making outputs non-reproducible
- Forgetting to update predictions after retraining
- Setting drift threshold too high, causing frequent unnecessary retrains
- Not modeling the time cost of training in realistic scenarios
Variations
- Use a rolling average of accuracy over multiple windows instead of a single point
- Trigger retraining based on feature distribution shift (PSI) rather than accuracy drop
Real-world use cases
- Monitoring production sentiment classifiers and automatically retraining on alert drift.
- Recommender systems detecting user behavior changes and refreshing model weights.
- Fraud detection models retraining on new patterns when performance dips below SLA.
Sponsored
More from ML engineering pipelines
- Bayesian Optimization in Python: A Simplified Mock Implementation medium
- Build a Data Helper Class in Python for ML Pipelines easy
- Build a Mock Random Forest Classifier in Python easy
- Champion Challenger Deployment Mock in Python easy
- Compare Model A vs Model B Metrics in Python easy
- Create a Minimal Great Expectations Suite Mock in Python easy
Keep learning
Related tutorials and quizzes for this topic.