How to Simulate a Micro-Batch Interval Trigger in Python
A dataclass-based mock that emits batch numbers at fixed intervals, mimicking a micro-batch streaming scheduler for testing and development.
Python code
32 linesimport time
from dataclasses import dataclass, field
from typing import List, Callable
@dataclass
class MicroBatchTriggerMock:
batch_interval_seconds: float = 0.5
max_batches: int = 5
_batches_emitted: int = 0
_next_emit_time: float = field(init=False, default=0)
def start(self, on_batch: Callable[[int], None]):
self._next_emit_time = time.time() + self.batch_interval_seconds
while self._batches_emitted < self.max_batches:
now = time.time()
if now >= self._next_emit_time:
self._batches_emitted += 1
on_batch(self._batches_emitted)
self._next_emit_time += self.batch_interval_seconds
else:
time.sleep(0.05)
def example_consumer(batch_number: int):
print(f"Batch {batch_number} processed at {time.time():.2f}")
if __name__ == "__main__":
trigger = MicroBatchTriggerMock(batch_interval_seconds=0.3, max_batches=3)
trigger.start(example_consumer)
print(f"Finished: {trigger._batches_emitted} batches emitted")
Output
Batch 1 processed at 1234567890.12
Batch 2 processed at 1234567890.42
Batch 3 processed at 1234567890.72
Finished: 3 batches emitted
How it works
The MicroBatchTriggerMock uses a time-based loop that checks if the next emit time has passed, then calls the callback with the batch number and advances the schedule by the interval. The field(init=False) ensures _next_emit_time is not a constructor argument but still part of the dataclass. The time.sleep(0.05) prevents busy-waiting while keeping timing accurate. This pattern is useful for emulating continuous data streams without needing a real message broker or scheduler.
Common mistakes
- Forgetting to reset `_batches_emitted` and `_next_emit_time` if reuse is needed
- Using `time.sleep(batch_interval_seconds)` which drifts with processing time
- Not handling interruption or cancellation for long-running loops
Variations
- Replace `time.sleep` with `asyncio.sleep` and convert to an async generator for concurrent pipelines
- Use a thread or process pool to emit batches in parallel with the consumer
Real-world use cases
- Unit-testing streaming consumers by generating controlled batch events without a live Kafka or Kinesis cluster.
- Prototyping backpressure and scheduling logic against fake time-based triggers before production integration.
- Creating deterministic data-generator tools that produce records at a fixed cadence for load or chaos testing.
Sponsored
More from Streaming & messaging
- At Most Once Fire-and-Forget Mock in Python easy
- Batch Consume Process Commit Pattern in Python medium
- Build a Streaming Messaging Helper in Python easy
- Dead Letter Queue Failed Messages List Mock in Python easy
- Dedupe processed message IDs in Python easy
- Event Envelope with Schema Version Field in Python easy
Keep learning
Related tutorials and quizzes for this topic.