FastAPI Background Tasks & Queues

Your API responds in milliseconds, but that email notification takes five seconds to send. Every slow task — image processing, PDF generation, webhook delivery — becomes a bottleneck that drags down latency and frustrates clients. The solution isn't to make your endpoint faster; it's to move work out of the request path entirely. In this lesson, you'll learn how to build background tasks and job queues in FastAPI, from the built-in BackgroundTasks for lightweight work to full job queues like Celery and RQ for heavy, durable workloads.

The problem this lesson solves

FastAPI runs your endpoint code synchronously by default — when a client calls your API, the server does everything before responding. If that includes sending an email, resizing an image, or hitting a third-party API, the client waits for all of it. This creates three painful consequences:

• Slow responses: Users see timeouts or spinner hell, even for simple actions like "create an account."
• Poor scalability: A burst of slow requests ties up server resources, starving fast endpoints.
• Lost transactions: If the request fails mid-task, the email never sends, the image never processes, and the client never knows.

The core problem: you're mixing request-critical work (validate the input, save the row, return 201) with side-effect work (notify someone, transform data, trigger a downstream system). These have completely different reliability and latency requirements.

Background tasks and job queues solve this by separating the two: the endpoint returns a quick acknowledgment, and the slow work runs after — either in-process after the response is sent, or in a separate worker process entirely.

Here's what you'll learn by the end of this lesson:

• How to run simple post-response work with FastAPI's built-in BackgroundTasks
• When that's enough — and when you need a real job queue like Celery or RQ
• How to design your tasks for retries, visibility, and failure
• How to troubleshoot the most common background-task pitfalls

Core concept / mental model

Think of your API as a restaurant. The host (your endpoint) takes the order, confirms it quickly, and sends the table to the kitchen. If the host personally cooked every meal before telling the customer the order is taken, everyone would starve. Instead, orders go on a ticket — a queue — and the kitchen (your workers) processes them asynchronously.

In FastAPI terms:

• The request/response cycle is the host — fast, short, and client-facing.
• The background task is a kitchen helper — it can do one or two quick prep steps after the host acknowledges.
• The job queue is the kitchen's ticket system — a durable list of orders that any cook (worker) can pick up, even if the first cook goes home.

The mental model rests on two key definitions:

• Background task: A function that runs after the HTTP response is sent, within the same process. FastAPI's built-in BackgroundTasks is perfect for chores that are quick, forgiving, and don't need to survive a server restart.
• Job queue (message broker + workers): A separate system (Redis, RabbitMQ) that stores jobs as messages. Workers — separate processes, often on different machines — consume those messages and execute the work. This gives you durability (jobs survive crashes), retries, parallelism, and scalability.

The critical distinction: a background task is fire-and-forget, while a job queue is fire-and-remember. If the server crashes after you queue a job with Celery, the job is still in Redis and will run. If it crashes before a BackgroundTasks function completes, that work is lost.

How it works step by step

Step 1: Using FastAPI's built-in BackgroundTasks

FastAPI ships with a lightweight solution: BackgroundTasks. You declare a parameter of type BackgroundTasks in your endpoint, add functions to it, and FastAPI runs them after the response is sent. Here's the flow:

1. Client hits /signup with a payload.
2. Your endpoint validates input and creates the user record.
3. You add a send_welcome_email function to background_tasks.
4. Your endpoint returns 201 Created immediately.
5. FastAPI sends the response, then executes send_welcome_email in the same process.

from fastapi import FastAPI, BackgroundTasks
from pydantic import BaseModel, EmailStr
import time

app = FastAPI()

def send_welcome_email(email: str):
    time.sleep(3)  # simulate slow third-party call
    print(f"Sending welcome email to {email}")

@app.post("/signup")
async def signup(user_data: UserCreate, background_tasks: BackgroundTasks):
    # 1. Validate and save user (simulated)
    user = {"email": user_data.email}
    # 2. Schedule background task
    background_tasks.add_task(send_welcome_email, user["email"])
    # 3. Return immediately
    return {"message": "User created", "id": 123}

Expected output (in the server log, after the response is sent):

INFO:     127.0.0.1:54321 - "POST /signup HTTP/1.1" 200 OK
Sending welcome email to user@example.com

Note: The response is sent before the sleep(3) finishes. The client sees a ~1ms response while the email takes 3 seconds in the background.

Step 2: When the built-in is not enough — job queues

BackgroundTasks is in-process — it shares memory with your API. If the API process restarts (deploy, crash), queued tasks vanish. For anything that needs guaranteed execution, retries, or horizontal scaling, you need a job queue.

The standard pattern:

1. Choose a message broker — Redis (simplest) or RabbitMQ (more features).
2. Define a task — a Python function decorated with @app.task (Celery) or @job (RQ).
3. Enqueue the task from your endpoint — the endpoint sends a message to the broker and returns immediately.
4. Run a worker process — a separate command starts workers that pull messages from the broker and execute tasks.

Here's a Celery example (install: pip install celery redis):

# tasks.py
from celery import Celery
import time

celery_app = Celery(
    "tasks",
    broker="redis://localhost:6379/0",
    backend="redis://localhost:6379/0"
)

@celery_app.task