Apache Kafka Fundamentals: A Practical Guide to Event-Driven and Distributed Systems

Before learning how Kafka works, it is important to understand the problems that led to the need for systems like Kafka. As applications grow into distributed systems, direct communication between services can create tight dependencies, performance problems, and failures that are difficult to handle. This lesson starts with those real-world challenges and gradually introduces asynchronous, event-driven communication as an approach to solving them.

The Distributed Systems Problem

Imagine an e-commerce system with several applications. The Order Service creates an order, the Payment Service processes the payment, the Inventory Service reserves products, and the Notification Service sends an email to the customer. These applications need to communicate, but they should not all depend directly on one another.

A simple approach is for one service to call another service through HTTP. For example, after creating an order, the Order Service might call the Payment Service directly:

POST /api/payments
{
  "orderId": 123,
  "amount": 150
}

This works well in a small system, but as the number of services grows, communication becomes increasingly complicated. The Order Service may need to know about Payment, Inventory, Shipping, Notifications, and Analytics. The result is a network of direct dependencies that becomes difficult to change and maintain.

Tight Coupling

Direct communication creates coupling between services. If the Order Service directly depends on the Payment Service, it needs to know where that service is, how its API works, and what response it expects.

Now imagine that another team wants to change the Payment API. They cannot safely make the change without considering all the applications that call it. As more services become connected, changing one service can require coordinating changes across many other services.

Kafka addresses this problem by introducing an intermediary. Instead of the Order Service directly calling every interested service, it can publish an event such as:

OrderCreated
{
    "orderId": 123,
    "amount": 150
}

Other services can independently receive that event and decide what to do with it. The Order Service does not need to know which services are interested in the event.

Synchronous Communication

HTTP APIs are commonly synchronous. The caller sends a request and normally waits for a response before continuing. This creates another problem in distributed systems.

Suppose the Order Service creates an order and then synchronously calls Payment, Inventory, Shipping, and Notification. If the Notification Service is temporarily unavailable, should the entire order operation fail? What happens if Payment takes five seconds to respond? What happens if Inventory is overloaded?

The original service becomes dependent on the availability and performance of all downstream services. Kafka allows many interactions to become asynchronous. The Order Service can publish an event and continue without waiting for every consumer to finish processing it.

Scalability

Communication can also become a scalability problem. Suppose an application receives 10,000 events per second. Sending every event directly to several independent services means each service must handle the incoming traffic independently, and the producer needs to manage many connections and failures.

Kafka provides a distributed event-streaming platform that can handle large volumes of events. Events can be distributed across multiple partitions, allowing consumers to process different parts of the workload concurrently. We will examine partitions and scalability in detail in later lessons.

Unreliable Communication

Distributed systems also fail in ways that do not happen inside a single application. A server can restart, a network connection can disappear, a service can become temporarily unavailable, or a deployment can interrupt communication.

With a simple HTTP call, an event can be lost if the caller cannot successfully deliver the request and there is no reliable retry or storage mechanism around it. Kafka provides durable storage for published records, allowing consumers to process events independently and, when appropriate, read them again later.

For example, if the Notification Service is temporarily unavailable after an OrderCreated event is published, Kafka can retain the event. When the Notification Service becomes available again, it can continue processing events rather than requiring the Order Service to recreate and resend them.

The Event-Driven Idea

This leads to the fundamental idea behind event-driven architecture. Instead of saying, "Service A, please execute this operation for me," a service can say, "Something happened."

For example, the Order Service publishes:

OrderCreated

The Payment Service can react by processing payment. The Inventory Service can react by reserving stock. The Notification Service can react by sending an email. An Analytics Service can react by recording the event for reporting.

The important point is that the producer does not need to know all of these consumers. This creates a more loosely coupled architecture where new consumers can be added without necessarily changing the original producer.

Where Kafka Fits

Kafka was designed to provide a scalable and durable platform for moving and storing streams of events between distributed applications. It is not simply an HTTP replacement, and thinking of Kafka as only a traditional message queue can lead to an incomplete understanding.

A useful mental model is to think of Kafka as a distributed event log. Producers write records to Kafka, Kafka stores those records, and consumers read them at their own pace. Because the records can remain available, consumers can process new events and, depending on the retention configuration, go back and process historical events as well.

In the next lessons, we will build this mental model step by step. We will first learn exactly what Kafka is and then examine the concepts that make this model work, including topics, partitions, producers, consumers, brokers, and offsets.