Fan Out Records to Multiple Sinks in Python

Distribute the same records across multiple target sinks (database, API, queue, etc.) using a defaultdict-based fan-out pattern.

Easy Python 3.8+ Aug 9, 2026 Data pipelines & processing 13 views 0 copies

fan-out defaultdict records pipelines distribute

Python code

27 lines

Python 3.8+

import json
from collections import defaultdict

SINKS = ["database", "api", "message_queue", "data_lake", "monitoring"]

def fan_out(records, *sinks):
    dist = defaultdict(list)
    for record in records:
        for sink in sinks:
            dist[sink].append(record)
    return dict(dist)

if __name__ == "__main__":
    records = [
        {"id": 1, "name": "Alice", "score": 95},
        {"id": 2, "name": "Bob", "score": 87},
        {"id": 3, "name": "Carol", "score": 92}
    ]
    
    result = fan_out(records, *SINKS)
    
    for sink, sink_records in result.items():
        print(f"{sink}: {len(sink_records)} records")
        for record in sink_records:
            print(f"  {json.dumps(record)}")
    
    print(f"\nTotal fan-out = {len(result) * len(records)} sink-record pairs")

Output

stdout

database: 3 records
  {"id": 1, "name": "Alice", "score": 95}
  {"id": 2, "name": "Bob", "score": 87}
  {"id": 3, "name": "Carol", "score": 92}
api: 3 records
  {"id": 1, "name": "Alice", "score": 95}
  {"id": 2, "name": "Bob", "score": 87}
  {"id": 3, "name": "Carol", "score": 92}
message_queue: 3 records
  {"id": 1, "name": "Alice", "score": 95}
  {"id": 2, "name": "Bob", "score": 87}
  {"id": 3, "name": "Carol", "score": 92}
data_lake: 3 records
  {"id": 1, "name": "Alice", "score": 95}
  {"id": 2, "name": "Bob", "score": 87}
  {"id": 3, "name": "Carol", "score": 92}
monitoring: 3 records
  {"id": 1, "name": "Alice", "score": 95}
  {"id": 2, "name": "Bob", "score": 87}
  {"id": 3, "name": "Carol", "score": 92}

Total fan-out = 15 sink-record pairs

How it works

The *sinks argument packaging collects any number of sink names passed after records, making the function flexible for 1–N destinations. defaultdict(list) auto-creates an empty list for each sink key on first access, so no manual initialization is needed. The nested loop iterates over each record and appends it to every sink's list, naturally duplicating data across targets. Returning dict(dist) converts the defaultdict back to a regular dict for a cleaner interface with standard dict behavior. The printed output shows each sink receiving all records, with the total confirming the expected fan-out multiplication.

Common mistakes

Using a regular dict without `.setdefault(sink, [])` before appending, causing KeyError at runtime.
Missing the `*` in `*sinks` when calling the function, accidentally passing a list as a single argument.
Reusing mutable record objects across sinks — if downstream processing mutates them, all copies change.

Variations

Yield each (sink, record) pair lazily with a generator and iterate in a downstream processing loop instead of accumulating everything in memory.
Use a dict comprehension with pre-built empty lists to avoid defaultdict if you prefer explicit initialization.

Real-world use cases

Replicating normalized events to a real-time dashboard API and a long-term analytics data lake.
Broadcasting the same user activity records to an audit log database and a message queue for spam scoring.
Duplicating incoming checkout payloads to payment processing, CRM sync, and a metrics endpoint.

Fan Out Records to Multiple Sinks in Python

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from Data pipelines & processing

Tutorials

Quizzes

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from Data pipelines & processing

Keep learning

Tutorials

Quizzes