Deploy Multi-Model Endpoints

You've trained a killer sentiment model and a separate NER model, and both need to live in production. Now what? Spin up two separate services, each with its own API, its own scaling rules, and its own deployment pipeline? That doubles your infrastructure cost, splits your monitoring, and turns every new model into another ops nightmare. This lesson shows you how to deploy multi-model endpoints — one HTTP endpoint, multiple models behind it — so you ship faster, scale smarter, and keep your ops burden flat as your model catalog grows.

The problem this lesson solves

Every ML model is easy to serve in isolation. The pain starts the moment you have more than one. You end up with a zoo of services, each with its own port, its own Docker image, its own autoscaling policy, and its own set of alerts. Your DevOps team is drowning in YAML, your cloud bill is climbing, and you're spending more time on plumbing than on improving your models.

The real pain points you'll face without a shared endpoint:

• Duplicate infrastructure — N models = N services = N times the memory, CPU, and cold-start latency.
• Fragmented observability — You have to stitch together logs and metrics from multiple services to understand user behavior.
• Slow rollout — Adding a new model means building a new service, wiring a new route, and updating the deployment pipeline.
• Resource contention — Models with bursty traffic sit idle next to models that are always busy, and you pay for the idle capacity.

By the end of this lesson, you'll be able to serve any number of models behind a single, clean Python HTTP interface, and you'll know exactly when it makes sense to do so.

Core concept / mental model

Think of your models as functions, not as services. When you deploy multi-model endpoints, you're building a single router that maps a request to the right function based on a simple key — typically the model name or ID.

REST API mental model: a multi-model endpoint is like a fast-food counter. The customer (your client) says "I want a burger" (model sentiment-v1) or "I want fries" (model ner-v2). The counter staff (your router) knows exactly which kitchen station to shout to. You don't build a separate restaurant for each dish.

Key terms you'll see in this lesson:

• Model registry — a catalog (dict, file, or database) that maps model IDs to loaded model objects.
• Router — the code that inspects the incoming request, finds the right model in the registry, and dispatches to it.
• Payload — the JSON body the client sends, which includes the model ID plus the actual input data.
• Inference — the act of running a prediction using a specific model.

The beauty of this design is that the client only ever talks to one URL. You can add, remove, or swap models behind that URL without breaking any client code.

How it works step by step

Let's trace the lifecycle of a single prediction request through a deployed multi-model endpoint.

1. Client sends a request — The client makes an HTTP POST to your endpoint, e.g., POST /predict. The JSON body includes a model field (e.g., "sentiment-v1") and a data field containing the actual input.

2. Your server receives it — A Python web framework like Flask or FastAPI grabs the JSON payload.

3. Router looks up the model — The server reads the model field and looks it up in a dictionary (the model registry). If the key exists, you get the loaded model object; if not, you return a 404 or 400 error.

4. Model runs inference — You call the model's predict method (or equivalent) with the input data.

5. Response is serialized and returned — The result is converted to JSON and sent back to the client.

Cause → effect chain: a clean request → fast lookup → predictable response. Every step is simple, testable, and easy to debug.

Hands-on walkthrough

A minimal Flask example

Let's start with a minimal but complete implementation using Flask. We'll create two dummy models (a sentiment scorer and an NER tagger) behind one endpoint.

# app.py
from flask import Flask, request, jsonify

# Dummy models
class SentimentModel:
    def predict(self, texts):
        return ["positive" if "good" in t else "negative" for t in texts]

class NERModel:
    def predict(self, texts):
        return [[{"word": w, "label": "ORG"} for w in t.split() if w.istitle()] for t in texts]

# Model registry
MODELS = {
    "sentiment-v1": SentimentModel(),
    "ner-v1": NERModel(),
}

app = Flask(__name__)

@app.route("/predict", methods=["POST"])
def predict():
    payload = request.get_json()
    model_id = payload.get("model")
    data = payload.get("data")

    model = MODELS.get(model_id)
    if model is None:
        return jsonify({"error": f"Unknown model: {model_id}"}), 404

    result = model.predict(data)
    return jsonify({"model": model_id, "result": result})

if __name__ == "__main__":
    app.run(port=8000)

To see it in action, run the server and send a few requests using curl:

curl -X POST http://localhost:8000/predict \
  -H "Content-Type: application/json" \
  -d '{"model": "sentiment-v1", "data": ["this is good", "this is bad"]}'

Expected output:

{"model":"sentiment-v1","result":["positive","negative"]}

Try the NER model too:

curl -X POST http://localhost:8000/predict \
  -H "Content-Type: application/json" \
  -d '{"model": "ner-v1", "data": ["Alice works at OpenAI", "Bob lives in Paris"]}'

Expected output:

{"model":"ner-v1","result":[[{"word":"Alice","label":"ORG"},{"word":"OpenAI","label":"ORG"}],[{"word":"Bob","label":"ORG"},{"word":"Paris","label":"ORG"}]]}

Pro tip: Using dependency injection for models makes it trivial to swap in real models (e.g., a Hugging Face transformer) for a production deployment.

FastAPI with async and validation

FastAPI brings automatic validation and async support, which becomes critical when you have models with different inference times. Here's a more robust version:

from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
from typing import List, Any

class PredictRequest(BaseModel):
    model: str
    data: List[Any]

class PredictResponse(BaseModel):
    model: str
    result: Any

MODELS = {
    "sentiment-v1": SentimentModel(),
    "ner-v1": NERModel(),
}

app = FastAPI()

@app.post("/predict", response_model=PredictResponse)
async def predict(req: PredictRequest):
    model = MODELS.get(req.model)
    if not model:
        raise HTTPException(status_code=404, detail=f"Unknown model: {req.model}")
    result = model.predict(req.data)
    return PredictResponse(model=req.model, result=result)

Now run it with uvicorn app:app --reload. The interactive docs at http://localhost:8000/docs let you test all your models with a single click.

Pro tip: Use FastAPI's async functions and CPU-bound model inference won't block the event loop — or offload heavy inference to a thread pool with run_in_executor.

Compare options / when to choose what

You have several ways to serve multiple models. Let's compare them head-to-head.

Simple rule of thumb: If you have fewer than, say, five models and your traffic is predictable, start with a single service. If you need independent scaling or your models are huge, split them. Model serving frameworks shine when your team has the expertise.

Troubleshooting & edge cases

AttributeError: 'NoneType' object has no attribute 'predict'

You passed a model ID that isn't in your registry. Check the spelling, and ensure the model is actually loaded.

# Bad
model = MODELS.get(payload.get("model"))
result = model.predict(data)  # AttributeError if model_id is misspelled

# Good
model = MODELS.get(payload.get("model"))
if model is None:
    return jsonify({"error": f"Model {model_id} not found"}), 404

Request body is None

If your API expects JSON but the client sends an empty body, request.get_json() can return None. Always validate the input.

if not payload or "model" not in payload or "data" not in payload:
    return jsonify({"error": "Missing 'model' or 'data' in JSON body"}), 400

One model throws an error; the whole endpoint dies

Wrap model-specific calls in try/except and return a structured error response so other models keep working.

try:
    result = model.predict(data)
except Exception as e:
    return jsonify({"error": f"Inference failed: {e}"}), 500

Cold start: model loading is slow

Heavy models can take seconds to load when the service starts. Options:

• Load models lazily (on first request) to speed up startup.
• Preload models in a worker process to avoid timeouts.
• Use a model registry service that keeps models loaded in memory.

What you learned & what's next

You've learned the core idea behind deploy multi-model endpoints: serve any number of models via a single HTTP API using a model registry and a router. You've built a working Flask example and a more robust FastAPI version, compared different deployment strategies, and debugged common pitfalls.

Key points to remember: - Multi-model endpoints reduce infrastructure to one service. - A model registry (dict) maps model IDs to model objects. - The client includes a model field in the JSON payload. - Always handle unknown models and malformed requests gracefully. - Choose single-service vs. multi-service based on traffic and scaling needs.

What's next: In your next lesson, you'll learn how to add authentication and rate limiting to your endpoints, securing them for production use. You'll build on the same pattern to make your API robust and production-ready.

Now go ahead and experiment — add a third model to your endpoint and try calling it. You'll see how easy scaling your model catalog becomes!