Token bucket rate limiter in Python (in-memory)

Implement a thread-safe in-memory token bucket rate limiter that throttles requests based on a steady refill rate.

Medium Python 3.9+ Aug 9, 2026 Reliability & rate limiting 14 views 0 copies

Python code

36 lines
Python 3.9+
import time
import threading


class TokenBucket:
    def __init__(self, capacity, refill_rate, refill_interval=1.0):
        self.capacity = capacity
        self.tokens = capacity
        self.refill_rate = refill_rate
        self.refill_interval = refill_interval
        self.last_refill = time.monotonic()
        self.lock = threading.Lock()

    def _refill(self):
        now = time.monotonic()
        elapsed = now - self.last_refill
        if elapsed > 0:
            new_tokens = elapsed * self.refill_rate
            self.tokens = min(self.capacity, self.tokens + new_tokens)
            self.last_refill = now

    def consume(self, tokens=1):
        with self.lock:
            self._refill()
            if self.tokens >= tokens:
                self.tokens -= tokens
                return True
            return False


if __name__ == "__main__":
    bucket = TokenBucket(capacity=5, refill_rate=2)
    for _ in range(8):
        print(f"consume 1 -> {bucket.consume()}")
    time.sleep(1)
    print(f"after 1s refill -> {bucket.consume()}")

Output

stdout
consume 1 -> True
consume 1 -> True
consume 1 -> True
consume 1 -> True
consume 1 -> True
consume 1 -> False
consume 1 -> False
consume 1 -> False
after 1s refill -> True

How it works

The token bucket algorithm works by maintaining a token count that increases over time at a fixed refill_rate. The _refill method calculates elapsed time using time.monotonic, adds the appropriate number of tokens, and caps the total at capacity to avoid overflow. consume acquires a lock to protect the token state for thread safety, ensuring multiple threads cannot accidentally over-spend tokens. In the example, the bucket starts full with 5 tokens so the first 5 consumes return True, then subsequent calls return False until enough time passes for a refill.

Common mistakes

  • Using `time.time()` instead of `time.monotonic()`, which can jump backward due to system clock adjustments.
  • Forgetting to cap tokens at capacity, causing the bucket to accumulate a large burst.
  • Not using a lock when accessing `tokens` from multiple threads, leading to race conditions.

Variations

  1. Use `threading.Condition` to block when tokens are unavailable instead of returning False.
  2. Create a decorator that applies the bucket to a function call.

Real-world use cases

  • Throttling outgoing API requests to stay within an external provider's quota.
  • Limiting login attempts per user to prevent brute-force attacks.
  • Capping click-through or event processing rates in a real-time pipeline.

Sponsored

Run this sample

Open the browser IDE to tweak the example and see results without installing anything.

Open editor

More from Reliability & rate limiting

Related tutorials and quizzes for this topic.