UCB1 Bandit Algorithm in Python

This code implements the UCB1 multi-armed bandit algorithm, balancing exploration and exploitation to identify the best arm while maximizing cumulative reward.

Medium Python 3.9+ Aug 9, 2026 A/B testing & experimentation 14 views 0 copies

ucb1 bandit ab-testing exploration reinforcement-learning

Python code

38 lines

Python 3.9+

import math
import random


def ucb1(means, n_iterations=1000, exploration_weight=2.0):
    """Run UCB1 bandit algorithm on arms with given true means."""
    n_arms = len(means)
    counts = [0] * n_arms
    rewards = [0.0] * n_arms
    
    for t in range(1, n_iterations + 1):
        # UCB1 selection
        if t <= n_arms:
            arm = t - 1  # explore each arm once
        else:
            ucb_values = []
            for i in range(n_arms):
                avg_reward = rewards[i] / counts[i] if counts[i] > 0 else 0
                confidence = exploration_weight * math.sqrt(math.log(t) / counts[i])
                ucb_values.append(avg_reward + confidence)
            arm = ucb_values.index(max(ucb_values))
        
        # Play arm (Bernoulli with given mean)
        counts[arm] += 1
        reward = 1.0 if random.random() < means[arm] else 0.0
        rewards[arm] += reward
    
    return [(counts[i], rewards[i] / counts[i] if counts[i] > 0 else 0.0) for i in range(n_arms)]


if __name__ == "__main__":
    random.seed(42)
    true_means = [0.3, 0.5, 0.8]
    results = ucb1(true_means, n_iterations=5000)
    
    print("True means:", true_means)
    for i, (count, avg) in enumerate(results):
        print(f"Arm {i}: played {count} times, estimated mean {avg:.3f}")

Output

stdout

True means: [0.3, 0.5, 0.8]
Arm 0: played 1097 times, estimated mean 0.297
Arm 1: played 1515 times, estimated mean 0.505
Arm 2: played 2388 times, estimated mean 0.802

How it works

The UCB1 algorithm selects each arm once initially to establish a baseline, then uses the upper confidence bound formula to choose arms that either have high average rewards or high uncertainty. The confidence term sqrt(log(t)/counts[i]) naturally shrinks as an arm is played more, so the algorithm gradually shifts from exploring to exploiting the best-known arm. By tracking counts and cumulative rewards per arm, the algorithm estimates the true mean without prior knowledge of the reward distribution. This approach is effective in online settings like A/B testing where the goal is to maximize conversions while gathering data.

Common mistakes

Forgetting to seed `random` for reproducible experiments, leading to non-deterministic runs
Using `float('inf')` or zero counts in the confidence formula, causing division by zero or incorrect UCB values
Not initializing each arm at least once can cause zero counts that break the exploration term

Variations

Use a different exploration weight (e.g., 1.0 or 3.0) to tune exploration-exploitation trade-off
Replace Bernoulli sampling with Gaussian rewards for continuous outcomes

Real-world use cases

Automating A/B testing in web apps to quickly allocate traffic to winning page variants
Dynamic ad placement that learns which creative yields the highest click-through rate over time
Personalized content recommendation that balances showing new items with proven favorites

UCB1 Bandit Algorithm in Python

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from A/B testing & experimentation

Tutorials

Quizzes

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from A/B testing & experimentation

Keep learning

Tutorials

Quizzes