Apache Kafka Fundamentals: A Practical Guide to Event-Driven and Distributed Systems

Now that we understand the problems Kafka is intended to address, we can look at Kafka itself and build a clear mental model of what it actually is. This lesson introduces Kafka's background, its origins at LinkedIn, and its main architectural components. The goal is to understand Kafka as a distributed event-streaming platform rather than simply thinking of it as another messaging queue.

The Story Behind Kafka

Apache Kafka started at LinkedIn around 2010 as an internal system for handling large amounts of activity data. LinkedIn had many different applications producing and consuming data, and the existing approaches were becoming difficult to scale. The engineers needed a system that could move large streams of data reliably between distributed applications while also keeping the data available for later consumption.

Kafka was later open-sourced and became an Apache project. Today, Apache Kafka is widely used in distributed systems, microservices, data integration, event-driven architecture, logging, analytics, and real-time data pipelines.

Kafka Is an Event-Streaming Platform

Kafka is best understood as a distributed event-streaming platform. An application called a producer publishes records to Kafka, Kafka stores those records, and applications called consumers read the records.

Consider an Order Service that creates an order. Instead of directly calling several other services, it can publish an event to Kafka:

OrderCreated
{
    "orderId": 1001,
    "customerId": 25,
    "amount": 150
}

The Payment Service can consume the event and process payment. The Inventory Service can consume the same event and reserve stock. An Analytics Service can also consume it and update reporting data. The Order Service does not need to know which of these services exist.

Kafka Is More Than a Message Queue

It is tempting to think of Kafka as simply another message queue such as RabbitMQ, but this mental model is incomplete. In a traditional queue, a message is commonly delivered to a consumer and then removed from the queue after successful processing. Kafka instead stores records in an ordered, persistent log, allowing consumers to track where they are and potentially read older records again.

For example, suppose an Analytics Service starts consuming OrderCreated events today. It can process new events as they arrive, but it can also potentially start from an earlier position and process historical events. This ability to retain and replay events is one of the characteristics that makes Kafka particularly useful for distributed event-driven systems.

Kafka's High-Level Architecture

At a high level, a Kafka system contains producers, brokers, topics, partitions, and consumers. Producers send records to topics. Kafka servers, called brokers, store those records. Consumers read records from topics.

You can visualize a simple system like this:

Order Service
     |
     | OrderCreated
     v
   Kafka
     |
     +----> Payment Service
     |
     +----> Inventory Service
     |
     +----> Analytics Service

A topic represents a stream of related records, such as orders. Kafka divides topics into partitions, which allow the data to be distributed across multiple brokers and processed concurrently. We will examine topics and partitions separately in upcoming lessons.

Kafka as a Distributed Log

The most useful mental model at this stage is to imagine Kafka as a distributed, persistent log of events. Producers append records to this log, while consumers independently read through it and keep track of their positions.

For example, the orders topic might contain:

Offset 0 -> OrderCreated 1001
Offset 1 -> OrderCreated 1002
Offset 2 -> OrderCreated 1003
Offset 3 -> OrderCreated 1004

A consumer can process these records and remember its position. If the consumer crashes after processing offset 2, it can later continue from an appropriate stored position. This design allows Kafka to separate storing events from processing events.

Where Kafka Fits in a Modern System

Kafka is particularly useful when many distributed applications need to exchange large volumes of events asynchronously. A typical architecture might contain ASP.NET Core microservices publishing events to Kafka, with other services consuming them. Kafka can also connect operational applications with databases, analytics platforms, data warehouses, and other external systems.

For example, a .NET application can use a Kafka client such as Confluent.Kafka to communicate with a Kafka cluster:

using Confluent.Kafka;

var producer = new ProducerBuilder<Null, string>(
    new ProducerConfig
    {
        BootstrapServers = "localhost:9092"
    }).Build();

await producer.ProduceAsync(
    "orders",
    new Message<Null, string>
    {
        Value = "OrderCreated:1001"
    });

The code is simple because the important complexity is inside Kafka itself. The producer sends a record to the orders topic, while Kafka handles storage, distribution, and other distributed-system responsibilities.

The key idea to remember from this lesson is that Kafka is a distributed platform for publishing, storing, and consuming streams of events. It provides a common backbone through which independent applications can exchange data without requiring every application to communicate directly with every other application.