Monitoring with Logging and Request IDs

You've deployed your FastAPI app, users are hitting it, and then it happens: a 500 error. Your logs are a wall of timestamps and error messages, but you have no idea which request failed, what the user was doing, or which logs belong to the same flow. You're not just debugging — you're spelunking through dark tunnels with a broken flashlight. This lesson is your headlamp: you'll learn how to implement monitoring with logging and request IDs so you can trace every request from the moment it hits your API to the final response, and turn your logs from noise into a detective's crime board.

The problem this lesson solves

Imagine a user reports that their order was charged twice but the confirmation page never loaded. You check the logs:

2025-01-15 14:32:01 ERROR: Payment gateway timeout
2025-01-15 14:32:01 INFO: Order created

Two separate events, no link. Was the timeout from the same request that created the order? You can't tell. Without a request ID — a unique identifier attached to every log entry for a single HTTP request — you're lost in a maze of unrelated timestamps.

The core problem: logs lack context. In a modern microservices or even a monolithic FastAPI app, a single user action triggers multiple database queries, external API calls, and background tasks. When something breaks, you need to answer three questions quickly: Which request failed? What did it do? What was the outcome? Without monitoring with logging and request IDs, answering these questions is guesswork — and in production, guessing costs you time and user trust.

Core concept / mental model

Think of a request ID as a correlation ID — a tracking number for your API request. Every log entry produced while handling that request carries the same ID. When you search your logs for that ID, you get the complete story of that request: every database query, every error, every response code.

Visualize it like a flight number. When you fly, every leg of your journey — check-in, security, boarding, the flight itself — is tied to that number. If something goes wrong, the airline can trace every step. Your request ID is that flight number for an HTTP request.

In implementation terms:

• Logging is the practice of recording events (info, warnings, errors) with structured data.
• Request ID is a unique identifier (usually a UUID) generated per incoming request, injected into the logging context, and included in every log record.
• Structured logging means emitting logs as JSON (or key-value pairs) so that machines can parse them and tools like Elasticsearch or Datadog can index them.

By combining these, you create traceable logs: every log line tells you what happened, when, and to which request.

How it works step by step

Here's the flow we'll implement:

1. Generate a request ID: When a new request arrives, create a unique ID (e.g., uuid4()) or reuse one from an incoming header (useful for distributed tracing across services).
2. Inject into logging context: Store the ID in a context variable that the logging system can access. In FastAPI, we often use contextvars or middleware to set a trace_id for the current request.
3. Log with context: Every log call inside the request handler includes the request ID automatically through a custom logging filter or a logger factory that reads the context variable.
4. Propagate to response: Send the request ID back in the response header (e.g., X-Request-ID) so the client can include it in support tickets.
5. Correlate across services: If your app calls external services, forward the request ID in outbound requests (like X-Request-ID), so the downstream logs can be tied together.

In FastAPI, middleware is the perfect place to generate and attach the ID, because it runs before the endpoint and after the response.

Hands-on walkthrough

Let's build a minimal but complete example. We'll use Python's built-in logging with a custom filter and a FastAPI middleware.

First, install FastAPI and uvicorn (if not already installed):

pip install fastapi uvicorn

Now create main.py:

import logging
import uuid
from contextvars import ContextVar
from fastapi import FastAPI, Request

# Context variable to hold the request ID for the current thread/async task
request_id_var: ContextVar[str] = ContextVar("request_id", default="-")

# Custom logging filter that injects the request ID into every log record
class RequestIDFilter(logging.Filter):
    def filter(self, record: logging.LogRecord) -> bool:
        record.request_id = request_id_var.get()
        return True

# Configure root logger
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger("app")
logger.addFilter(RequestIDFilter())

# Format with request ID
formatter = logging.Formatter("%(asctime)s | %(levelname)s | request_id=%(request_id)s | %(message)s")
handler = logging.StreamHandler()
handler.setFormatter(formatter)
logger.handlers = [handler]
logger.propagate = False

app = FastAPI()

@app.middleware("http")
async def add_request_id(request: Request, call_next):
    # Generate or reuse a request ID
    request_id = request.headers.get("X-Request-ID", str(uuid.uuid4()))
    request_id_var.set(request_id)
    logger.info("Request started")
    response = await call_next(request)
    response.headers["X-Request-ID"] = request_id
    logger.info("Request finished")
    return response

@app.get("/items/{item_id}")
async def get_item(item_id: int):
    logger.info(f"Fetching item {item_id}")
    # Simulate some work
    return {"item_id": item_id}

Run with:

uvicorn main:app --reload

Here's a sample log output when you hit http://localhost:8000/items/42:

2025-01-15 15:01:02 | INFO | request_id=1f2c9d8e-... | Request started
2025-01-15 15:01:02 | INFO | request_id=1f2c9d8e-... | Fetching item 42
2025-01-15 15:01:02 | INFO | request_id=1f2c9d8e-... | Request finished

Notice every log line shares the same request_id. If you send a second request, you'll see a different ID, allowing you to filter logs per request.

For structured logging (JSON), you can use python-json-logger:

pip install python-json-logger

Update the handler:

from pythonjsonlogger import jsonlogger

handler = logging.StreamHandler()
handler.setFormatter(jsonlogger.JsonFormatter("%(asctime)s %(levelname)s %(request_id)s %(message)s"))
logger.handlers = [handler]

Now logs look like:

{"asctime": "...", "levelname": "INFO", "request_id": "...", "message": "Fetching item 42"}

This JSON format is what log aggregators (ELK, Datadog, etc.) love.

Compare options / when to choose what

When to use what

• If you're just starting, use the built-in logging filter approach — it's transparent and teaches you the mechanics.
• As your app grows, switch to structlog to get JSON logs, automatic request context, and easy integration with log aggregation.
• If you need distributed tracing across multiple services, consider OpenTelemetry — it extends request IDs into full traces (spans, parent/child relationships) and integrates with many backends.

Troubleshooting & edge cases

Missing request ID in logs

You see %s like request_id=% instead of the actual value. This happens when the filter isn't applied to the logger or the context variable is empty. Ensure you add the filter to the logger you use (not just the root logger), and that the middleware sets the variable before any log call.

Race conditions in async code

ContextVar is the right choice for async, but if you use global variables or threading.local, you'll get wrong IDs in concurrent requests. Always use ContextVar in async contexts.

Client-provided request ID

If you trust a client-supplied X-Request-ID, you risk log injection (e.g., malicious strings). Sanitize it: validate it matches a UUID pattern or generate a new one if invalid. Example:

import uuid
from fastapi import HTTPException

def validate_request_id(req_id: str) -> str:
    try:
        uuid.UUID(req_id)
        return req_id
    except ValueError:
        raise HTTPException(status_code=400, detail="Invalid X-Request-ID")

Log noise

Too much logging kills performance and storage. Log meaningful events (request start, errors, external calls) and use DEBUG level sparingly. Use log levels strategically: INFO for user actions, WARNING for abnormal but recoverable situations, ERROR for failures.

Duplicate logs

Make sure your logger doesn't propagate to the root logger if you already added a handler; otherwise you'll see duplicate lines. Set logger.propagate = False as in the example.

What you learned & what's next

You now understand monitoring with logging and request IDs: why it matters, how to implement it in FastAPI using middleware and contextvars, and how to make logs structured and traceable. You can generate a request ID per request, inject it into all log records, and pass it back to the client. You also know when to choose different logging libraries and how to avoid common pitfalls like race conditions and log noise.

In the next lesson, you'll build on this foundation to add structured error tracking with tools like Sentry — automatically capturing exceptions with rich context (including your request ID) so you can debug production issues even faster.

Keep your logs clean, your IDs consistent, and your debugging fast. That's the power of monitoring with logging and request IDs.