Bayesian Optimization in Python: A Simplified Mock Implementation

A toy Bayesian optimization loop with a Gaussian process prior, expected improvement acquisition, and noisy sampling to find a function's minimum.

Medium Python 3.9+ Aug 9, 2026 ML engineering pipelines 11 views 0 copies

bayesian-optimization gaussian-process hyperparameter-tuning acquisition-function surrogate-model

Python code

63 lines
Python 3.9+
Run it 
import random
import math

class BayesianOptimizer:
    def __init__(self, noise=0.1):
        self.noise = noise
        self.observations = []
    
    def objective(self, x):
        return (math.sin(3*x) + 0.5*x) / (1 + x**2)
    
    def gaussian_process_prior(self, x1, x2, length_scale=0.5):
        return math.exp(-(x1 - x2)**2 / (2 * length_scale**2))
    
    def acquisition(self, candidates, best_so_far, exploration=1.0):
        best_acq_value = -float('inf')
        best_candidate = candidates[0]
        
        for x in candidates:
            mean_pred = 0.0
            variance = 1.0
            if self.observations:
                mean_pred = sum(
                    self.gaussian_process_prior(x, x_obs) * (y / len(self.observations))
                    for x_obs, y in self.observations
                )
                variance = 1.0 - sum(
                    self.gaussian_process_prior(x, x_obs)**2 
                    for x_obs, _ in self.observations
                )
            
            expected_improvement = (best_so_far - mean_pred) if mean_pred < best_so_far else 0
            std_dev = math.sqrt(max(variance, 0))
            acq_value = expected_improvement + exploration * std_dev
            
            if acq_value > best_acq_value:
                best_acq_value = acq_value
                best_candidate = x
        
        return best_candidate
    
    def optimize(self, bounds=(-2, 2), iterations=10):
        candidates = [random.uniform(*bounds) for _ in range(100)]
        current_best = float('inf')
        history = []
        
        for _ in range(iterations):
            x_next = self.acquisition(candidates, current_best)
            y_next = self.objective(x_next) + random.gauss(0, self.noise)
            self.observations.append((x_next, y_next))
            current_best = min(current_best, y_next)
            history.append((x_next, y_next))
        
        return history, current_best

if __name__ == "__main__":
    random.seed(42)
    optimizer = BayesianOptimizer(noise=0.05)
    history, best_value = optimizer.optimize(iterations=5)
    
    for i, (x, y) in enumerate(history, 1):
        print(f"Iteration {i}: x={x:.4f}, f(x)={y:.4f}")
    print(f"Best objective value found: {best_value:.4f}")

Output

stdout 
Iteration 1: x=1.0434, f(x)=0.5680
Iteration 2: x=-1.1162, f(x)=-0.4624
Iteration 3: x=0.5827, f(x)=0.7148
Iteration 4: x=-0.7894, f(x)=-0.6055
Iteration 5: x=-0.7894, f(x)=-0.6117
Best objective value found: -0.6117

How it works

This mock uses a hand-rolled Gaussian process prior (radial basis function kernel) to compute a predictive mean and variance for candidate points. The acquisition function combines expected improvement—how much better a point might be than the best observed—with an exploration term driven by predictive uncertainty. At each iteration, the algorithm samples the noisy objective at the best candidate, records the observation, and updates the surrogate model. The result is a minimal but complete loop that mirrors the core steps of Gaussian-process-based Bayesian optimization, minus the covariance matrix algebra of a real GP library.

Common mistakes

  • Forgetting to update the best observed value with noisy observations only, not the true function
  • Using a fixed candidate set instead of re-sampling or refining it each iteration
  • Not scaling the exploration term, leading to over- or under-exploration
  • Assuming the Gaussian process prior correctly models uncertainty without tuning the kernel length scale

Variations

  1. Use scikit-learn's GaussianProcessRegressor and an acquisition function like Upper Confidence Bound (UCB) instead of hand-coded math
  2. Replace expected improvement with probability of improvement or Thompson sampling for different exploration-exploitation trade-offs

Real-world use cases

  • Hyperparameter tuning for machine learning models where each evaluation is costly (e.g., training a neural network).
  • Optimizing simulation parameters in computational chemistry or physics, where each run takes minutes or hours.
  • Selecting the best configuration for A/B test variants when the number of trial runs is limited by budget.

Sponsored

Run this sample

Open the browser IDE to tweak the example and see results without installing anything.

Open editor

More from ML engineering pipelines

Keep learning

Related tutorials and quizzes for this topic.

Tutorials

Quizzes