Apache Kafka Fundamentals: A Practical Guide to Event-Driven and Distributed Systems

Kafka introduces several terms that appear repeatedly in almost every Kafka-based system, and understanding these terms is essential before moving into more advanced topics. This lesson walks through producers, consumers, topics, partitions, brokers, clusters, records, and offsets, showing how they fit together. By the end, you should be able to look at a basic Kafka architecture and understand what each component is responsible for.

The Kafka Mental Model

Before writing Kafka applications, you need to understand its vocabulary. Imagine an Order Service producing OrderCreated events, Kafka storing those events, and several services consuming them. Kafka uses a small set of concepts to describe this entire flow: producers, consumers, topics, partitions, brokers, clusters, records, and offsets.

The relationship can be summarized as follows:

Producer
   |
   | Records
   v
Topic
   |
   +-- Partition 0
   +-- Partition 1
   +-- Partition 2
   |
Kafka Cluster
   |
   +--> Consumer
   +--> Consumer

Producer

A producer is an application that publishes records to Kafka. For example, your ASP.NET Core Order Service can act as a producer when it publishes an OrderCreated event.

Using the .NET Confluent.Kafka client, a producer might look like this:

await producer.ProduceAsync(
    "orders",
    new Message<Null, string>
    {
        Value = "OrderCreated:1001"
    });

The producer does not normally send the record directly to another application. Instead, it sends the record to a Kafka topic.

Consumer

A consumer is an application that reads records from Kafka. For example, a Payment Service can consume records from the orders topic and react whenever an order is created.

The important distinction is that the consumer controls its reading position. Kafka does not simply push a record to a consumer and immediately forget about it. The consumer keeps track of which records it has processed through offsets, which we will examine later.

Record

A record is the individual piece of data stored in Kafka. You can think of it as an event or message containing information such as a key, value, timestamp, and other metadata.

For example:

Key:   order-1001
Value: OrderCreated
       orderId = 1001
       amount = 150

The record is the basic unit that producers write and consumers read.

Topic

A topic is a named stream of records. You can think of it as a logical category of events. For example, an application might have topics called orders, payments, and shipments.

When the Order Service publishes an OrderCreated record, it might publish it to the orders topic:

Order Service
      |
      | OrderCreated
      v
   "orders"

A topic does not represent one particular consumer. Multiple independent consumers can read from the same topic, which is important when several services need the same event.

Partition

A partition is a subdivision of a Kafka topic. Kafka does not necessarily store a topic as one large sequence of records. Instead, a topic can contain multiple partitions.

For example:

orders topic

Partition 0: Order 1001, Order 1004
Partition 1: Order 1002, Order 1005
Partition 2: Order 1003, Order 1006

Partitions allow Kafka to distribute data and processing across multiple servers and consumers. They are also important for ordering because Kafka guarantees ordering within a partition, not across all partitions of a topic.

Broker

A broker is a Kafka server responsible for storing and serving records. A Kafka environment normally contains multiple brokers rather than a single server.

For example:

Kafka Cluster

Broker 1
  orders - Partition 0

Broker 2
  orders - Partition 1

Broker 3
  orders - Partition 2

Multiple brokers allow Kafka to distribute data and provide fault tolerance. If one broker fails, replicated partitions can allow the system to continue operating.

Cluster

A Kafka cluster is a group of Kafka brokers working together. From the application's perspective, the cluster provides a single Kafka environment even though the data may be distributed across several servers.

For example, a production system might have three or more brokers. The producers and consumers connect to the cluster, while Kafka manages where individual partitions are stored.

Offset

An offset identifies the position of a record within a partition. Consider this partition:

Offset 0 -> Order 1001
Offset 1 -> Order 1002
Offset 2 -> Order 1003
Offset 3 -> Order 1004

If a consumer has successfully processed offset 2, it knows where it is in the partition. If the application restarts, it can continue from an appropriate offset instead of necessarily starting from the beginning.

This is one of the major differences between Kafka and a simple queue. The record remains in Kafka according to the topic's retention policy, while consumers maintain their own progress through the stream.

Putting Everything Together

Consider an Order Service publishing an event:

Order Service
   |
   | Producer
   v
Kafka Cluster
   |
   | orders topic
   |
   +-- Partition 0
   +-- Partition 1
   +-- Partition 2
   |
   +--> Payment Service
   |       Consumer
   |
   +--> Inventory Service
           Consumer

The Order Service is the producer. The OrderCreated event is a record. The orders stream is a topic. The topic is divided into partitions. Kafka servers are brokers, and the collection of brokers forms a cluster. Payment and Inventory are consumers, and each consumer uses offsets to track its progress.

Once these relationships are clear, Kafka becomes much easier to understand. The next important question is why Kafka needs partitions in the first place and how partitions provide both scalability and ordering.