How to Simulate a Micro-Batch Interval Trigger in Python

A dataclass-based mock that emits batch numbers at fixed intervals, mimicking a micro-batch streaming scheduler for testing and development.

Easy Python 3.9+ Aug 9, 2026 Streaming & messaging 13 views 0 copies

Python code

32 lines
Python 3.9+
import time
from dataclasses import dataclass, field
from typing import List, Callable


@dataclass
class MicroBatchTriggerMock:
    batch_interval_seconds: float = 0.5
    max_batches: int = 5
    _batches_emitted: int = 0
    _next_emit_time: float = field(init=False, default=0)

    def start(self, on_batch: Callable[[int], None]):
        self._next_emit_time = time.time() + self.batch_interval_seconds
        while self._batches_emitted < self.max_batches:
            now = time.time()
            if now >= self._next_emit_time:
                self._batches_emitted += 1
                on_batch(self._batches_emitted)
                self._next_emit_time += self.batch_interval_seconds
            else:
                time.sleep(0.05)


def example_consumer(batch_number: int):
    print(f"Batch {batch_number} processed at {time.time():.2f}")


if __name__ == "__main__":
    trigger = MicroBatchTriggerMock(batch_interval_seconds=0.3, max_batches=3)
    trigger.start(example_consumer)
    print(f"Finished: {trigger._batches_emitted} batches emitted")

Output

stdout
Batch 1 processed at 1234567890.12
Batch 2 processed at 1234567890.42
Batch 3 processed at 1234567890.72
Finished: 3 batches emitted

How it works

The MicroBatchTriggerMock uses a time-based loop that checks if the next emit time has passed, then calls the callback with the batch number and advances the schedule by the interval. The field(init=False) ensures _next_emit_time is not a constructor argument but still part of the dataclass. The time.sleep(0.05) prevents busy-waiting while keeping timing accurate. This pattern is useful for emulating continuous data streams without needing a real message broker or scheduler.

Common mistakes

  • Forgetting to reset `_batches_emitted` and `_next_emit_time` if reuse is needed
  • Using `time.sleep(batch_interval_seconds)` which drifts with processing time
  • Not handling interruption or cancellation for long-running loops

Variations

  1. Replace `time.sleep` with `asyncio.sleep` and convert to an async generator for concurrent pipelines
  2. Use a thread or process pool to emit batches in parallel with the consumer

Real-world use cases

  • Unit-testing streaming consumers by generating controlled batch events without a live Kafka or Kinesis cluster.
  • Prototyping backpressure and scheduling logic against fake time-based triggers before production integration.
  • Creating deterministic data-generator tools that produce records at a fixed cadence for load or chaos testing.

Sponsored

Run this sample

Open the browser IDE to tweak the example and see results without installing anything.

Open editor

More from Streaming & messaging

Related tutorials and quizzes for this topic.