How to Create a Mock Kafka Producer in Python

Build a Kafka producer that generates mock streaming records with JSON serialization and error handling for local testing.

Easy Python 3.9+ Aug 9, 2026 Big data & Spark 16 views 0 copies

Requires third-party packages — install first
pip install kafka-python

Python code

37 lines
Python 3.9+
import json
import time
from kafka import KafkaProducer
from kafka.errors import KafkaError

def create_mock_producer(bootstrap_servers="localhost:9092", topic="input-topic"):
    """Create a Kafka producer that generates mock streaming data."""
    producer = KafkaProducer(
        bootstrap_servers=bootstrap_servers,
        value_serializer=lambda v: json.dumps(v).encode("utf-8")
    )

    def send_mock_record(record_id):
        mock_record = {
            "id": record_id,
            "timestamp": time.time(),
            "value": record_id * 10,
            "status": "active" if record_id % 2 == 0 else "inactive"
        }
        try:
            future = producer.send(topic, value=mock_record)
            future.get(timeout=10)
            print(f"Sent: {json.dumps(mock_record)}")
        except KafkaError as e:
            print(f"Failed to send record {record_id}: {e}")

    return send_mock_record

if __name__ == "__main__":
    send_mock = create_mock_producer()

    # Simulate streaming a few records
    for i in range(1, 6):
        send_mock(i)
        time.sleep(0.5)

    print("Mock streaming completed.")

Output

stdout
Sent: {"id": 1, "timestamp": 1699876543.123456, "value": 10, "status": "inactive"}
Sent: {"id": 2, "timestamp": 1699876543.623456, "value": 20, "status": "active"}
Sent: {"id": 3, "timestamp": 1699876544.123456, "value": 30, "status": "inactive"}
Sent: {"id": 4, "timestamp": 1699876544.623456, "value": 40, "status": "active"}
Sent: {"id": 5, "timestamp": 1699876545.123456, "value": 50, "status": "inactive"}
Mock streaming completed.

How it works

The KafkaProducer is configured with a value_serializer that converts each dict into a UTF-8 encoded JSON string, so the broker receives clean JSON payloads. The inner send_mock_record closure captures the producer and topic, returning a reusable function that sends one record at a time. Calling future.get(timeout=10) blocks until the broker acknowledges the write, making failures visible immediately. The try/except around KafkaError catches connection issues or serialization problems without crashing the whole stream. This pattern is useful for testing Spark Structured Streaming jobs against a lightweight local Kafka before wiring up real data sources.

Common mistakes

  • Forgetting to call `.get(timeout=...)` on the future, which silently drops failed sends
  • Using `json.dumps` without encoding, causing TypeError when Kafka tries to serialize bytes
  • Hardcoding the topic name instead of passing it as a parameter to the producer factory

Variations

  1. Use `producer.send(topic, key=record_id, value=mock_record)` to set a partition key for ordered processing
  2. Switch to `confluent-kafka` for a C-backed producer with lower latency in production

Real-world use cases

  • Simulating user clickstream events to validate a Spark Structured Streaming aggregation pipeline before production rollout.
  • Generating test telemetry for an IoT ingestion service, ensuring downstream consumers handle bursts of JSON messages.
  • Creating synthetic order data to benchmark Kafka consumer throughput and tune batch sizes in a data platform.

Sponsored

Run locally

This sample needs third-party packages, so it cannot run in the browser IDE. Copy the code above, install the packages shown at the top, then run it in your own Python environment.

More from Big data & Spark

Related tutorials and quizzes for this topic.