Count Records Processed per Category in Python

Use a Counter dictionary to track how many records of each type (ok, error, retry) were processed in a data pipeline.

Easy Python 3.9+ Aug 9, 2026 Data pipelines & processing 15 views 0 copies

counter metrics data-pipeline collections counter-per-category

Python code

14 lines

Python 3.9+

from collections import Counter
import random

processed_counter = Counter()

def process_records(records):
    for record in records:
        processed_counter[record] += 1
    return len(records)

if __name__ == "__main__":
    sample_records = [random.choice(["ok", "error", "retry"]) for _ in range(10)]
    print(f"Processed {process_records(sample_records)} records")
    print("Counter:", dict(processed_counter))

Output

stdout

Processed 10 records
Counter: {'ok': 4, 'error': 3, 'retry': 3}

How it works

The Counter class from the collections module provides an efficient way to count hashable items. Incrementing with processed_counter[record] += 1 works even when the key doesn't exist because Counter defaults missing keys to zero. The function returns the total count of records processed while the counter updates in place, making it easy to emit metrics at the end of a pipeline. Since the sample uses random choices, the exact output varies, but the structure remains consistent.

Common mistakes

Using a plain dict and forgetting to check if the key exists before incrementing
Confusing the return value of the function with the contents of the counter
Resetting the counter inside the loop instead of once at the start

Variations

Use a defaultdict(int) instead of Counter for the same counting behavior
Pass the counter as an argument to keep the function pure and testable

Real-world use cases

Tracking success and failure counts per batch in an ETL job to emit to monitoring dashboards.
Counting log levels (INFO, WARN, ERROR) across records in a processing queue for observability.
Measuring retry attempts versus final outcomes in a message pipeline to tune backoff policies.

Count Records Processed per Category in Python

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from Data pipelines & processing

Tutorials

Quizzes

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from Data pipelines & processing

Keep learning

Tutorials

Quizzes