Apache Kafka Fundamentals: A Practical Guide to Event-Driven and Distributed Systems

The concepts of topics and partitions are at the center of how Kafka stores and distributes data. A topic may look like a simple named stream from the application's perspective, but internally Kafka divides that stream into partitions to achieve scalability and parallel processing. This lesson explores why partitions exist, how they are distributed, and why Kafka's ordering guarantee is limited to individual partitions.

What Is a Topic?

A topic is a named stream where Kafka stores related records. You can think of a topic as a logical category of events. For example, an e-commerce system might have an orders topic for order-related events and a payments topic for payment-related events.

When the Order Service creates an order, it can publish an event to the orders topic:

OrderCreated: 1001
OrderCreated: 1002
OrderCreated: 1003

The topic separates one type of event stream from another. However, Kafka does not normally store all records of a topic on one server. This is where partitions become important.

Why Does Kafka Use Partitions?

Imagine that the orders topic receives one million events per second. If all those events had to be stored and processed by a single Kafka server, that server would eventually become a bottleneck. Kafka solves this by dividing the topic into multiple partitions.

For example:

orders topic

Partition 0: Order 1001, Order 1004, Order 1007
Partition 1: Order 1002, Order 1005, Order 1008
Partition 2: Order 1003, Order 1006, Order 1009

Each partition is an ordered sequence of records. Kafka can distribute these partitions across different brokers in a cluster. This means multiple servers can store and serve different parts of the same topic.

Partitions Provide Scalability

Partitions allow Kafka to scale horizontally. Suppose one broker cannot handle the workload of the orders topic. You can have multiple brokers, with different partitions distributed across them.

The same idea applies to consumers. If a topic has three partitions, a consumer group can have up to three active consumers processing those partitions concurrently:

Partition 0 ---> Consumer 1
Partition 1 ---> Consumer 2
Partition 2 ---> Consumer 3

If your application needs more processing capacity, you can add consumer instances, provided there are enough partitions to distribute the work. Therefore, the number of partitions is an important architectural decision because it affects potential parallelism.

Ordering Within a Partition

Kafka guarantees that records are ordered within a partition. Consider this partition:

Partition 0

Offset 0 -> OrderCreated 1001
Offset 1 -> OrderUpdated 1001
Offset 2 -> OrderShipped 1001

A consumer reading this partition sees the records in that order. This is useful when the sequence of events matters.

However, Kafka does not guarantee a single global ordering across multiple partitions. Consider:

Partition 0: OrderCreated 1001, OrderUpdated 1001

Partition 1: OrderCreated 1002, OrderUpdated 1002

Kafka cannot guarantee whether a consumer processing both partitions will observe the events from partition 0 before or after the events from partition 1. Each partition has its own independent ordering.

Why Not One Global Order?

The reason is closely related to scalability. If Kafka had to maintain one global ordered sequence for an entire topic, all producers and consumers would effectively depend on that single sequence. This would make parallel processing much more difficult.

Partitions allow Kafka to split the workload while preserving ordering where it matters. For example, an application might require all events for the same order to be processed in sequence, but it does not necessarily need every order in the entire system to be globally ordered.

This is why choosing a good record key is important. Kafka can use a key such as orderId to consistently place events for the same order into the same partition:

Key: order-1001
      |
      v
Partition 2

OrderCreated
OrderPaid
OrderShipped

The events for order-1001 can therefore maintain their relative order while other orders are processed in parallel.

A topic represents a logical stream of related events, while a partition is an independently ordered portion of that stream. Kafka uses partitions primarily to distribute data and processing across multiple brokers and consumers.

The most important rule to remember is: Kafka guarantees ordering within a partition, not across an entire topic. This trade-off between ordering and parallelism is fundamental to Kafka's distributed design.