Kafka Message Production

You've built services that send HTTP requests and read from databases, but now you need to decouple your producers from your consumers so that a spike in traffic doesn't crash your backend. In this lesson, you'll learn how to produce messages to Kafka with confluent-kafka, the industry-standard Python client that gives you low-level control, high throughput, and the reliability guarantees your systems need. By the end, you'll be able to send hundreds of thousands of messages per second with just a few lines of code and debug the common pitfalls that trip up every Kafka beginner.

The problem this lesson solves

Kafka is a distributed log, not a traditional message queue. With RabbitMQ or Redis, each message is often consumed and deleted. With Kafka, every message is appended to a partition and kept for a configured retention period, allowing multiple independent consumer groups to read the same data at their own pace. The challenge is that this model requires a different mental approach: you don't just "send a message"; you produce a record to a topic, and the broker decides which partition it lands on.

Without a solid producer setup, you'll face message loss, duplicate messages, or unbounded latency during peak load. For example, a naive producer that doesn't handle broker-side errors might silently drop messages when a partition leader is temporarily unavailable. This lesson gives you a battle-tested recipe for building a reliable producer with confluent-kafka, the library that powers many of the world's largest data pipelines.

By the time you finish, you'll be able to:

• Create a producer instance with sensible defaults
• Send both keyed and unkeyed messages
• Handle delivery reports to ensure messages actually reached the broker
• Tune batching and acknowledgments for your throughput/durability tradeoff

Core concept / mental model

Think of Kafka as a gigantic, immutable append-only file distributed across multiple machines. A producer's job is to write a new line (record) to that file. The file is split into partitions, each of which is an ordered sequence. When you send a message with a key, a hash of that key determines the partition, so all messages with the same key end up in the same partition — preserving order for that key. Without a key, messages are distributed round-robin (or based on the partitioner's stickiness), which maximizes throughput but sacrifices per-key ordering.

The producer does not send one message at a time. Instead, it buffers messages in memory and sends them in batches to the broker. This is why confluent-kafka can achieve such high throughput — network round-trips are amortized over thousands of records. The key configuration options are:

• batch.size: number of bytes to accumulate before sending
• linger.ms: how long to wait for more messages before sending a batch
• acks: how many replicas must acknowledge the write

Here's a mental picture of the flow:

Your app → Producer.buffer → [batch.size, linger.ms] → Network → Kafka broker (acks=1/all) → check delivery report

When you call produce(), the message doesn't go to the broker immediately; it goes into the producer's internal queue. The flush() method blocks until all queued messages are sent and their delivery reports are processed. This is the fundamental mental model: produce is async, flush is sync.

How it works step by step

Now let's walk through the mechanics of producing messages to Kafka with confluent-kafka in a logical, cause-and-effect sequence.

1. Install the library

First, you need to install confluent-kafka:

pip install confluent-kafka

This library wraps the C++ librdkafka, which is why it's so fast and battle-tested. No additional system dependencies are needed for basic usage.

2. Create a producer configuration

Your producer needs at minimum a bootstrap.servers list so it knows where to find brokers. You also want to set client.id to identify your app in broker logs, and acks to control durability.

from confluent_kafka import Producer

conf = {
    'bootstrap.servers': 'localhost:9092',
    'client.id': 'python-producer',
    'acks': 'all',  # wait for all replicas to ack
}
producer = Producer(conf)

3. Send messages with delivery callbacks

The produce() method takes a topic, a value (bytes or string), an optional key, and a callback. The callback is invoked when the broker acknowledges the record (or when an error occurs). This is your safety net — it lets you know if a message was lost.

def delivery_callback(err, msg):
    if err:
        print(f'Message failed: {err}')
    else:
        print(f'Message delivered to {msg.topic()} [{msg.partition()}] at offset {msg.offset()}')

producer.produce('orders', key='user_42', value='order-123-placed', callback=delivery_callback)

4. Flush and poll

After sending messages, you must call flush() to ensure everything is sent. Under the hood, flush() also triggers the library to process delivery reports. In a long-running producer, you should periodically call producer.poll(0) to process those callbacks without blocking.

producer.flush()  # blocks until all messages are delivered

Hands-on walkthrough

Let's build a complete, runnable example. First, ensure you have Kafka running locally (e.g., via Docker Compose). Then create a topic named orders with one partition for simplicity.

# Start Kafka with Docker
docker run -d -p 9092:9092 --name kafka apache/kafka:3.7.0
# Create a topic
docker exec kafka kafka-topics --create --topic orders --partitions 3 --replication-factor 1 --bootstrap-server localhost:9092

Now, write a producer script producer.py:

from confluent_kafka import Producer
import json
import time

conf = {
    'bootstrap.servers': 'localhost:9092',
    'client.id': 'order-producer',
    'acks': 'all',
}
producer = Producer(conf)

orders = [
    {'user_id': 42, 'item': 'book', 'qty': 1},
    {'user_id': 43, 'item': 'laptop', 'qty': 2},
    {'user_id': 42, 'item': 'pen', 'qty': 5},
]

def delivery_callback(err, msg):
    if err:
        print(f'Delivery failed for {msg.topic()}: {err}')
    else:
        print(f'Delivered to {msg.topic()} partition {msg.partition()} offset {msg.offset()}')

for order in orders:
    # Use user_id as key to keep each user's orders in order
    key = str(order['user_id'])
    value = json.dumps(order).encode('utf-8')
    producer.produce('orders', key=key, value=value, callback=delivery_callback)
    # Non-blocking poll to handle callbacks
    producer.poll(0)

# Wait for all messages to be delivered
producer.flush()
print('All messages sent.')

Run it:

python producer.py

Expected output (order may vary):

Delivered to orders partition 0 offset 0
Delivered to orders partition 2 offset 0
Delivered to orders partition 0 offset 1
All messages sent.

Pro tip: Always call flush() before your program exits. Otherwise, you may lose buffered messages that were never sent.

Sending high-throughput messages

If you need to send thousands of messages per second, avoid calling flush() or poll() in a tight loop. Instead, let the batch fill up and rely on linger.ms to control latency. Here's a batch producer:

from confluent_kafka import Producer
import time

conf = {
    'bootstrap.servers': 'localhost:9092',
    'acks': '1',  # only leader ack — faster but less durable
    'linger.ms': 10,
    'batch.size': 16384,
}
producer = Producer(conf)

for i in range(10000):
    producer.produce('events', value=f'event-{i}'.encode())
    if i % 1000 == 0:
        producer.poll(0)  # process delivery reports occasionally

producer.flush()

Compare options / when to choose what

Confluent-Kafka isn't the only way to produce messages to Kafka from Python. Here's a quick comparison:

• confluent-kafka is the recommended choice for real-world production. It's the same library used by many Fortune 500 companies and is actively maintained by Confluent.
• kafka-python is fine for learning, but you'll quickly hit performance walls and lag behind on protocol features.
• Faust adds a stream-processing layer on top of confluent-kafka, but it's an extra dependency and might be overkill if you just need a simple producer.

Rule of thumb: If you care about throughput, durability, and production readiness, use confluent-kafka. For a hurried prototype, kafka-python will do.

Troubleshooting & edge cases

1. Local: Queue full or Message timed out

When your producer sends more messages than the broker can handle, the internal buffer fills up and produce() raises BufferError. This usually means:

• You forgot to call poll() or flush(), so delivery reports are never processed and buffers never free up.
• The broker is slow or down, and message.timeout.ms is too low.

Fix: Call producer.poll(0) periodically in a loop, and consider increasing queue.buffering.max.messages or message.timeout.ms.

2. Local: Broker transport failure

The producer cannot reach any broker. This is a network issue, not a code issue. Check your bootstrap.servers address and ensure Kafka is running.

Fix: Verify with nc -zv localhost 9092 and check Kafka logs.

3. Delivery callback never fires

If you never see callbacks, you probably forgot to call poll() or flush(). In confluent-kafka, callbacks are only triggered when the internal poller runs.

Fix: Call producer.poll(0) in your main loop or at least before exiting.

4. Ordering surprises with multiple partitions

If you don't specify a key or you use a key with low cardinality, messages with the same key can still go to different partitions if the partitioner changes. Also, when a partition fails over, the producer may retry and potentially reorder messages — only idempotence=true guarantees exactly-once ordering.

Fix: Use idempotence=true in your config (available in confluent-kafka ≥1.4).

What you learned & what's next

You now know how to produce messages to Kafka with confluent-kafka — from the underlying mental model to a complete runnable producer. You can configure batching, acknowledgments, and delivery callbacks to fit your reliability and performance needs. You've also seen how to handle the most common errors.

As a next step, you'll learn how to consume those messages with confluent-kafka's high-level consumer, where you'll deal with offsets, consumer groups, and rebalancing. That's the other half of the puzzle — building a reliable data pipeline is about both ends.

Now, practice by producing messages to a topic of your own and observing the delivery reports. Then, be ready to flip the switch and start reading them back.