Kafka Topics & Partitions

You’ve built queues that deliver a message once, maybe twice, and you’ve felt the pain of a slow consumer dragging everything down. Now imagine a system where millions of events per second must be ingested, replayed, and processed by multiple independent teams — a plain queue collapses under that load. This is the problem Kafka solves, and at the heart of that solution are two deceptively simple concepts: topics and partitions. By the end of this lesson, you’ll not only understand these building blocks but also be able to design a Kafka topic layout that scales with your workload.

The problem this lesson solves

Traditional message brokers like RabbitMQ or Amazon SQS treat a queue as a single, ordered pipeline. That model has a critical limitation: throughput is capped by the processing speed of a single consumer. If you need to process 100,000 messages per second and your consumer can only handle 1,000, you’re stuck — you can add more consumers, but the queue itself doesn’t parallelize. You also face the rebalancing nightmare of competing consumers fighting over a shared log.

Kafka throws that model away. Instead of a queue, it uses a distributed commit log — an immutable, ordered sequence of records. But even a log has limits. A single log file on a single disk can only sustain so many writes and reads. To scale, Kafka splits each log into partitions, which are stored across multiple brokers. This is the leap that lets Kafka handle billions of events per day: parallelism is built into the data model, not bolted on.

Without understanding topics and partitions, you’ll make one of two mistakes: you’ll treat a topic like a queue and wonder why you can’t scale consumers, or you’ll over-partition and watch your cluster choke on metadata. This lesson gives you the mental model to avoid both.

Core concept / mental model

Think of a topic as a named category in a database — like a users table or an orders file. It groups related events, such as “user-clicks” or “payment-transactions.” Producers write to a topic; consumers read from it. But unlike a database table, a topic is not a single file. It’s a logical umbrella over one or more partitions.

A partition is an ordered, immutable sequence of records. It’s the smallest unit of parallelism in Kafka. Here’s the key: ordering is guaranteed only within a partition, never across partitions. If you need events for a specific user to be processed in order, you must ensure they all land in the same partition — usually by using the user ID as the partition key.

Picture a topic called orders with 3 partitions. It’s like having three separate queues, each with its own complete, ordered list of orders. Producers distribute orders across these three queues; consumers can each grab one queue and process in parallel. But if the same order appears in two different queues, you lose cross-queue ordering.

Key definitions:

• Broker: A Kafka server that stores partitions and serves clients.
• Offset: A monotonically increasing integer assigned to each record within a partition. It’s the consumer’s bookmark.
• Replication: Each partition can be copied across brokers for fault tolerance (not the focus here, but essential in production).
• Consumer group: A set of consumers that collaboratively read from a topic’s partitions — each partition is assigned to exactly one consumer in the group.

The mental model that sticks: topic = a folder; partition = a file inside that folder. The folder can have many files; files can be hosted on different machines; you get parallel reads by splitting work across files.

How it works step by step

Let’s trace the life of a message from producer to consumer, focusing on how partitions come into play.

Step 1: Producer sends a record

A producer creates a record with a key (optional) and a value (the payload). For example, key="user_123", value="{\"click\":\"button\"}". The producer asks Kafka: “Which partition should this go to?” If a key is provided, Kafka hashes it (using the default partitioner, which does a hash modulo the partition count) to pick a partition. If no key is given, records are distributed in a round-robin fashion to balance load.

Step 2: Kafka appends to a partition

Once a partition is chosen, the broker appends the record to the end of that partition’s log and assigns it a monotonically increasing offset. The offset is unique within that partition — record 0, 1, 2, and so on. This offset becomes the consumer’s pointer.

Step 3: Consumer group reads

Consumers subscribe to the topic with a group ID. The Kafka coordinator assigns each consumer a subset of the partitions. In the simplest case, one consumer gets all partitions; with more consumers, the partitions are spread out. The consumer reads records sequentially from each assigned partition, starting at a given offset (e.g., from the beginning or from the last committed offset).

Step 4: Ordering and parallelism

Because each partition is ordered and independent, a consumer can process records from multiple partitions in any order, but within a partition, order is preserved. To get both parallelism and ordering, you must carefully choose your partition key. If you want all events for user_123 in order, use user_123 as the key — that guarantees they all go to the same partition.

This is the entire magic of Kafka: partition count determines the maximum parallelism. If you have 3 partitions, you can have up to 3 consumers in a group, each reading one partition. More consumers than partitions means some consumers sit idle.

Hands-on walkthrough

Let’s get our hands dirty. For this exercise, assume you have a local Kafka instance running (e.g., via Docker Compose or a standalone download). We’ll use the kafka-topics.sh CLI and a small Python producer/consumer with confluent-kafka.

Step 1: Create a topic with multiple partitions

Open a terminal and run:

# Create a topic named 'orders' with 3 partitions and replication factor 1 (for local dev)
kafka-topics.sh --bootstrap-server localhost:9092 \
  --create --topic orders \
  --partitions 3 --replication-factor 1

Check the topic description:

kafka-topics.sh --bootstrap-server localhost:9092 \
  --describe --topic orders

You’ll see output like:

Topic: orders   PartitionCount: 3   ReplicationFactor: 1
    Topic: orders   Partition: 0    Leader: 1   Replicas: 1 Isr: 1
    Topic: orders   Partition: 1    Leader: 1   Replicas: 1 Isr: 1
    Topic: orders   Partition: 2    Leader: 1   Replicas: 1 Isr: 1

That confirms three partitions exist.

Step 2: Produce records with and without keys

Now produce some records. First, without a key — using a simple bash producer:

# Produce 5 records without a key (round-robin)
for i in 1 2 3 4 5; do echo "order-$i" | kafka-console-producer.sh \
  --bootstrap-server localhost:9092 --topic orders; done

Now produce with a key, so all records for the same key land in the same partition:

# Produce 3 records with key 'user_123'
echo "user_123:order-A" | kafka-console-producer.sh \
  --bootstrap-server localhost:9092 --topic orders --property parse.key=true --property key.separator=:
echo "user_123:order-B" | kafka-console-producer.sh \
  --bootstrap-server localhost:9092 --topic orders --property parse.key=true --property key.separator=:
echo "user_123:order-C" | kafka-console-producer.sh \
  --bootstrap-server localhost:9092 --topic orders --property parse.key=true --property key.separator=:

Step 3: Consume from the beginning and see partition assignment

Use the console consumer to read all records from the start:

kafka-console-consumer.sh --bootstrap-server localhost:9092 \
  --topic orders --from-beginning --property print.partition=true

You’ll see output similar to:

Partition:0 offset:0    order-1
Partition:1 offset:0    order-2
Partition:2 offset:0    order-3
Partition:0 offset:1    order-4
Partition:1 offset:1    order-5
Partition:0 offset:2    user_123:order-A
Partition:0 offset:3    user_123:order-B
Partition:0 offset:4    user_123:order-C

Notice how the unkeyed records were spread across partitions (0,1,2), but all keyed records with user_123 went to partition 0. That’s the partitioner at work.

Step 4: Python consumer example

Let’s write a Python script that consumes from our topic, using the confluent-kafka library:

# consumer.py
from confluent_kafka import Consumer, KafkaError

conf = {
    'bootstrap.servers': 'localhost:9092',
    'group.id': 'order-processors',
    'auto.offset.reset': 'earliest'
}
consumer = Consumer(conf)
consumer.subscribe(['orders'])

try:
    while True:
        msg = consumer.poll(1.0)
        if msg is None:
            continue
        if msg.error():
            if msg.error().code() == KafkaError._PARTITION_EOF:
                continue
            else:
                print(f"Consumer error: {msg.error()}")
                break
        print(f"partition={msg.partition()} offset={msg.offset()} key={msg.key()} value={msg.value()}")
except KeyboardInterrupt:
    pass
finally:
    consumer.close()

Run it:

pip install confluent-kafka
python consumer.py

You’ll see each record’s partition and offset. If you start a second consumer with the same group ID, the partitions will be split between them — that’s Kafka’s built-in load balancing.

Compare options / when to choose what

Now that you’ve seen partitions in action, let’s compare key decisions: partition count, key usage, and consumer group size.

Pro tip: Partition count is fixed at topic creation. You can increase it later, but records may be re-partitioned, breaking per-key ordering for existing keys. Choose based on peak expected throughput and future growth, but don’t over-partition — each partition adds metadata and file-handle overhead.

Alternatives to explore:

• Keyless partitions: Good for log aggregation where order doesn’t matter; round-robin gives balanced load. This is the simplest approach.
• Custom partitioner: You can implement a custom class in Java or Python to route records based on business logic (e.g., data locality, hot-key avoidance), but the default hash is usually fine.
• Compact topics: Topics with a cleanup.policy=compact retain only the latest value for each key — useful for state stores. This is a variation you’ll likely meet later.

Troubleshooting & edge cases

Here are the most common pitfalls when working with topics and partitions:

1. More consumers than partitions

You add 5 consumers to a topic with 3 partitions. Expect 2 consumers idle. Fix: increase partitions or reduce consumers. You can check consumer group status with kafka-consumer-groups.sh --describe --group your-group.

2. Losing order for keyed records

You switch from 3 to 5 partitions. A key that used to land in partition 0 now goes to partition 2, and you lose the global ordering for that key. Fix: never increase partitions if strict per-key ordering is a cross-partition guarantee. Plan capacity from day one.

3. Hot partitions

All your records use a small set of keys (e.g., a few celebrity users). Their partitions get far more traffic, causing bottlenecks. Fix: add a salt to the key (e.g., user_123_0, user_123_1) to spread load — but know that ordering for that user across partitions is lost.

4. Consumer offset resetting unexpectedly

New consumer group starts from latest offset and skips existing records. Fix: set auto.offset.reset=earliest when you want to read from the beginning. Check your consumer config.

5. Producer errors about unknown topic

If a topic doesn’t exist, Kafka may auto-create it with default partition count (often 1). Fix: explicitly create topics with desired partitions; disable auto-creation in production.

What you learned & what's next

You now understand the core of Kafka’s scalability: topics organize data streams, and partitions enable parallel consumption. You’ve seen how keys determine partitioning and ordering, and you’ve practiced creating topics, producing messages, and consuming with both CLI and Python. You also know the critical trade-offs: partition count vs. consumer parallelism, and the dangers of over-partitioning or key mishandling.

This is the foundation for everything else in Kafka — consumer groups, offset management, replication, and exactly-once semantics all build on this mental model. In the next lesson, you’ll dive into consumer groups and offset management, learning how to commit offsets, handle rebalances, and guarantee delivery semantics. With topics and partitions mastered, you’re ready to build production-grade event pipelines.