Broadcast a Small Reference Table in Python

Simulates SQL-style broadcasting of a small lookup table against a larger fact table in memory for mockups or load tests.

Easy Python 3.9+ Aug 9, 2026 Database scaling & optimization 14 views 0 copies

broadcast mock-data data-engineering reference-table

Python code

30 lines

Python 3.9+

import random

def broadcast_mock(target, source, columns):
    result = {}
    for col in columns:
        if col in target and col in source:
            result[col] = target[col] + [source[col][i % len(source[col])] for i in range(len(target[col]))]
        elif col in target:
            result[col] = target[col]
        elif col in source:
            result[col] = [source[col][i % len(source[col])] for i in range(len(target[next(iter(target))]))]
    return result

if __name__ == "__main__":
    target = {
        "id": [1, 2, 3, 4],
        "value": [10, 20, 30, 40]
    }
    source = {
        "region": ["North", "South"],
        "multiplier": [1.5, 2.0]
    }

    broadcasted = broadcast_mock(target, source, ["id", "region", "multiplier"])
    print(broadcasted)

    max_len = max(len(v) for v in broadcasted.values())
    for i in range(max_len):
        row = {k: v[i] if i < len(v) else None for k, v in broadcasted.items()}
        print(row)

Output

stdout

{'id': [1, 2, 3, 4], 'region': ['North', 'South', 'North', 'South'], 'multiplier': [1.5, 2.0, 1.5, 2.0]}
{'id': 1, 'region': 'North', 'multiplier': 1.5}
{'id': 2, 'region': 'South', 'multiplier': 2.0}
{'id': 3, 'region': 'North', 'multiplier': 1.5}
{'id': 4, 'region': 'South', 'multiplier': 2.0}

How it works

The broadcast_mock function mimics how a database would repeat rows from a small reference table to match the row count of a larger table. For each column in the source, it uses source[col][i % len(source[col])] to cycle through the reference values for every target row. When a column exists only in the source, it still broadcasts it by matching the length of any target column. This pattern is handy for creating realistic mock data without loading a full RDBMS.

Common mistakes

Assuming the reference table and target table always share a key column.
Using a fixed index into the source list without modulo, which causes IndexError when source is shorter than target.
Forgetting to handle the case where a target column is empty, leading to division by zero in the modulo.
Not using `len(target[next(iter(target))])` carefully when target may be empty.

Variations

Use `itertools.cycle` from the standard library to cycle through reference values instead of manual modulo arithmetic.
Pre-zip the target and broadcasted source columns into tuples using `zip` and list comprehensions for a more functional style.

Real-world use cases

Creating a realistic fact table for database load testing by repeating small dimension rows across many records.
Building mock API responses that emulate joined data for front-end prototyping without a backend.
Generating synthetic datasets for ML model training where categorical features repeat across samples.

Broadcast a Small Reference Table in Python

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from Database scaling & optimization

Tutorials

Quizzes

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from Database scaling & optimization

Keep learning

Tutorials

Quizzes