How to Create a Mock Kafka Producer in Python
Build a Kafka producer that generates mock streaming records with JSON serialization and error handling for local testing.
pip install kafka-python
Python code
37 linesimport json
import time
from kafka import KafkaProducer
from kafka.errors import KafkaError
def create_mock_producer(bootstrap_servers="localhost:9092", topic="input-topic"):
"""Create a Kafka producer that generates mock streaming data."""
producer = KafkaProducer(
bootstrap_servers=bootstrap_servers,
value_serializer=lambda v: json.dumps(v).encode("utf-8")
)
def send_mock_record(record_id):
mock_record = {
"id": record_id,
"timestamp": time.time(),
"value": record_id * 10,
"status": "active" if record_id % 2 == 0 else "inactive"
}
try:
future = producer.send(topic, value=mock_record)
future.get(timeout=10)
print(f"Sent: {json.dumps(mock_record)}")
except KafkaError as e:
print(f"Failed to send record {record_id}: {e}")
return send_mock_record
if __name__ == "__main__":
send_mock = create_mock_producer()
# Simulate streaming a few records
for i in range(1, 6):
send_mock(i)
time.sleep(0.5)
print("Mock streaming completed.")
Output
Sent: {"id": 1, "timestamp": 1699876543.123456, "value": 10, "status": "inactive"}
Sent: {"id": 2, "timestamp": 1699876543.623456, "value": 20, "status": "active"}
Sent: {"id": 3, "timestamp": 1699876544.123456, "value": 30, "status": "inactive"}
Sent: {"id": 4, "timestamp": 1699876544.623456, "value": 40, "status": "active"}
Sent: {"id": 5, "timestamp": 1699876545.123456, "value": 50, "status": "inactive"}
Mock streaming completed.
How it works
The KafkaProducer is configured with a value_serializer that converts each dict into a UTF-8 encoded JSON string, so the broker receives clean JSON payloads. The inner send_mock_record closure captures the producer and topic, returning a reusable function that sends one record at a time. Calling future.get(timeout=10) blocks until the broker acknowledges the write, making failures visible immediately. The try/except around KafkaError catches connection issues or serialization problems without crashing the whole stream. This pattern is useful for testing Spark Structured Streaming jobs against a lightweight local Kafka before wiring up real data sources.
Common mistakes
- Forgetting to call `.get(timeout=...)` on the future, which silently drops failed sends
- Using `json.dumps` without encoding, causing TypeError when Kafka tries to serialize bytes
- Hardcoding the topic name instead of passing it as a parameter to the producer factory
Variations
- Use `producer.send(topic, key=record_id, value=mock_record)` to set a partition key for ordered processing
- Switch to `confluent-kafka` for a C-backed producer with lower latency in production
Real-world use cases
- Simulating user clickstream events to validate a Spark Structured Streaming aggregation pipeline before production rollout.
- Generating test telemetry for an IoT ingestion service, ensuring downstream consumers handle bursts of JSON messages.
- Creating synthetic order data to benchmark Kafka consumer throughput and tune batch sizes in a data platform.
Sponsored
More from Big data & Spark
Keep learning
Related tutorials and quizzes for this topic.