Handle Model Inference Errors

Your RAG pipeline just returned a 500 to a paying customer because the LLM API hiccuped — or worse, your batch job silently wrote None into a database column and corrupted your analytics. Inference errors are not a matter of if; they're a matter of when. Rate limits, timeouts, content filters, and malformed responses happen daily in production. In this lesson, you'll learn to handle model inference errors gracefully — turning catastrophic failures into recoverable events with retries, fallbacks, and structured logging.

The problem this lesson solves

LLM APIs are powerful but notoriously unreliable. Even with a stable internet connection, you can hit:

• Rate limits (HTTP 429) — your app exceeded the provider's quota, and the API refuses more requests.
• Timeouts (e.g., requests.exceptions.ConnectTimeout) — the model took too long to generate, and your HTTP client gave up.
• Content filter blocks — the provider's safety layer rejected your prompt, returning a generic error.
• Service outages (HTTP 5xx) — the provider's infrastructure is having a bad day.
• Malformed JSON responses — the model hallucinated a response that breaks your schema.

If you don't handle these gracefully, your application fails loudly: users see 500 errors, batch jobs crash, and logs become ciphers. The cost isn't just downtime — it's lost trust and wasted compute. This lesson teaches you a systematic approach to catch, classify, retry, and degrade when inference fails.

Why it matters now: By lesson 137, you've built features around LLM APIs. Without error handling, your app is a fragile demo. With it, you build production-grade resilience — the difference between a hobby script and a reliable service.

Core concept / mental model

Think of inference error handling as defensive driving around the API. You can't control the road (the provider's status), but you can control your car (your code). The mental model has three layers:

1. Detection — recognize what went wrong (timeout? rate limit? content filter? validation error?) by inspecting exception types and response metadata.
2. Response — act on that classification with a strategy: retry for transient errors, fallback to a smaller model for capacity issues, or degrade gracefully for permanent blocks.
3. Observation — log every failure with enough context to debug later, without leaking sensitive prompt data.

Think of it like a circuit breaker in electrical engineering: when the current spikes, the breaker trips to protect the system. In your app, when the API starts failing repeatedly, you open the circuit — pause requests, maybe return a cached response — to avoid hammering a broken endpoint.

Key definitions:

• Transient error — temporary condition (network blip, rate limit). Retrying often succeeds.
• Permanent error — unfixable by retry (malformed prompt, invalid API key). Retrying is wasteful.
• Retry with exponential backoff — retry with increasing wait times (e.g., 1s, 2s, 4s) to avoid hammering the server.
• Fallback model — a cheaper or smaller model (e.g., gpt-3.5-turbo instead of gpt-4) used when the primary is unavailable.
• Graceful degradation — return a default value or a slimmed-down response instead of an exception.

How it works step by step

Here's the logical flow you'll implement in your error-handling wrapper:

1. Wrap the call — put the inference call inside a try/except block to catch exceptions.
2. Classify the error — inspect the exception type and HTTP status code. Is it transient (Timeout, 429, 5xx) or permanent (InvalidRequest, AuthError)?
3. Retry with backoff — for transient errors, retry a limited number of times (say 3) with exponential backoff, including a small random jitter.
4. Fallback — after retries fail, try an alternative model or an offline heuristic (e.g., return a canned response).
5. Log and alert — log the error type, message, retry count, and response latency. If it's critical, send an alert to your monitoring system.
6. Return something safe — if all else fails, return a default fallback value that your application can handle, rather than letting an exception crash the app.

Each step is cause → effect: you classify correctly → you retry only when useful → you avoid wasted compute → you preserve user experience.

Hands-on walkthrough

Let's build a robust call_model function that demonstrates graceful error handling. We'll use Python's requests library against a mock OpenAI-style endpoint, but the same pattern applies to any LLM provider.

First, define a custom exception and a retry wrapper:

import time
import random
import requests
from requests.exceptions import Timeout

class InferenceError(Exception):
    """Custom exception for model inference failures."""
    def __init__(self, message, retryable=False):
        super().__init__(message)
        self.retryable = retryable

def classify_status(status_code):
    """Map HTTP status to a retryable flag."""
    if status_code in (429, 500, 502, 503, 504):
        return True
    return False

def call_model(prompt, api_key, max_retries=3):
    """Call an LLM API with retries and exponential backoff."""
    url = "https://api.example.com/v1/completions"
    headers = {"Authorization": f"Bearer {api_key}", "Content-Type": "application/json"}
    payload = {"prompt": prompt, "max_tokens": 50}

    for attempt in range(max_retries + 1):
        try:
            response = requests.post(url, headers=headers, json=payload, timeout=10)
            if response.status_code == 200:
                return response.json()["choices"][0]["text"]
            else:
                if classify_status(response.status_code):
                    raise InferenceError(f"HTTP {response.status_code}: {response.text}", retryable=True)
                else:
                    raise InferenceError(f"HTTP {response.status_code}: {response.text}", retryable=False)
        except Timeout as e:
            raise InferenceError(f"Connection timeout: {e}", retryable=True)
        except requests.RequestException as e:
            raise InferenceError(f"Request failed: {e}", retryable=False)
        except InferenceError as e:
            if e.retryable and attempt < max_retries:
                wait = (2 ** attempt) + random.uniform(0, 1)
                time.sleep(wait)
                continue
            raise
    # Should never reach here, but good practice
    raise InferenceError("Exhausted retries", retryable=False)

# Example usage
try:
    result = call_model("What's the capital of France?", api_key="sk-test")
    print(f"Result: {result}")
except InferenceError as e:
    print(f"Inference failed: {e}")

Expected output (if the mock API fails with 429 twice then succeeds):

Attempt 1: 429, retrying in ~1s
Attempt 2: 429, retrying in ~3s
Result: Paris

Now let's add a fallback model and a graceful degradation path. Suppose you use the openai library; here's a more production-worthy pattern:

import openai
from tenacity import retry, wait_exponential, stop_after_attempt, retry_if_exception_type

class ModelRouter:
    def __init__(self, primary_model="gpt-4", fallback_model="gpt-3.5-turbo", default_response="I'm sorry, I couldn't process that."):
        self.primary = primary_model
        self.fallback = fallback_model
        self.default = default_response

    @retry(
        wait=wait_exponential(multiplier=1, min=1, max=10),
        stop=stop_after_attempt(3),
        retry=retry_if_exception_type((openai.APITimeoutError, openai.RateLimitError))
    )
    def _call(self, model, messages):
        client = openai.OpenAI()  # assumes API key in env
        resp = client.chat.completions.create(model=model, messages=messages)
        return resp.choices[0].message.content

    def generate(self, messages):
        try:
            return self._call(self.primary, messages)
        except (openai.APIConnectionError, openai.APIStatusError, openai.AuthenticationError):
            print("Primary failed, falling back to", self.fallback)
            try:
                return self._call(self.fallback, messages)
            except Exception as e:
                print(f"Fallback failed: {e}")
                return self.default

# Usage
router = ModelRouter()
response = router.generate([{"role": "user", "content": "Hello!"}])
print(response)

Expected output (if primary times out but fallback succeeds):

Primary failed, falling back to gpt-3.5-turbo
Hello! How can I assist you today?

Finally, logging errors with structure helps debugging. Use Python's logging module:

import logging

logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
logger = logging.getLogger(__name__)

def log_inference_failure(error, context):
    logger.error("Inference failed: %s | context: %s", error, context, exc_info=True)
# Call this in your except block instead of print.

Now you have a complete, reproducible example you can adapt to your provider.

Compare options / when to choose what

Not all error-handling strategies are equal. Here's a comparison to help you choose:

When to choose what:

• If your app is a prototype, start with a simple try/except.
• If you're in production and the API is generally stable, add retries with backoff.
• If you need high availability (e.g., customer-facing chat), combine retries + fallback model + graceful degradation.
• If you're doing batch processing, consider a circuit breaker to pause the queue when the provider fails repeatedly.

For most cases in this track, a retry + fallback combination is the sweet spot.

Troubleshooting & edge cases

Here are common pitfalls and fixes:

• Infinite retry loops: Without retry caps, your app can hang indefinitely. Always set max_retries and use exponential backoff with a ceiling.
• Retrying on permanent errors: Don't retry on authentication failures (401) or invalid requests (400). You'll waste time and quota. Use classification to filter.
• Not handling JSON decoding errors: The model may return malformed JSON. Wrap response.json() in a try/except and have a fallback parser.
• Leaking sensitive data in logs: Never log the raw prompt or API keys. Log only error types, status codes, and a hashed prompt ID.
• Using print for production logging: Print statements don't show timestamps or levels. Use the logging module.
• Timeouts too short: If you set timeout=1 for a complex model, you'll get false timeouts. Tune it based on expected latency.

If you see RateLimitError despite retries, check your quota and consider a queue to space out requests. If you see APIConnectionError, check your network and proxy settings.

What you learned & what's next

You've learned to handle model inference errors gracefully: you can classify errors, retry with backoff, fall back to alternative models, and log failures contextually. You can now build LLM applications that don't crash on the first hiccup — a crucial skill for production AI engineering.

Next up: In the next lesson, you'll measure model provider reliability and design fallback strategies around latency and cost. You'll apply these error-handling patterns to build a multi-provider aggregator, taking your app from resilient to intelligent.