Apache Kafka Fundamentals: A Practical Guide to Event-Driven and Distributed Systems

Once we understand where Kafka stores events, the next question is how applications actually put those events there. This lesson follows the journey of a record from an application into Kafka and examines the decisions made along the way, including keys, serialization, partition selection, and acknowledgments. A practical .NET example will make these concepts concrete and show how a real application can publish an event.

What Is a Kafka Producer?

A producer is an application that creates and publishes records to Kafka. In a microservices system, an ASP.NET Core service can act as a producer whenever it needs to publish an event for other applications.

Consider an Order Service. When a customer places an order, the service can publish an OrderCreated event instead of directly calling the Payment and Inventory services. The producer's responsibility is to create the record, determine where it should go, serialize its data, and send it to Kafka.

Records, Keys, and Values

A Kafka record commonly contains a key and a value. The key identifies the entity associated with the event, while the value contains the actual event data.

For example:

Key:   order-1001
Value: OrderCreated
       orderId = 1001
       amount = 150

The key is particularly important because Kafka can use it to determine the partition. If every event for order-1001 uses the same key, Kafka can place those events in the same partition, helping preserve their ordering.

Serialization

Applications work with objects such as C# classes, but Kafka stores and transfers bytes. Serialization converts an application object into a format that can be transmitted and stored, while deserialization performs the reverse operation.

For example, an Order Service might have:

public record OrderCreated(
    int OrderId,
    decimal Amount);

The object could be serialized to JSON:

{
  "orderId": 1001,
  "amount": 150
}

In production systems, JSON, Avro, or Protobuf can be used depending on the requirements. The important idea is that both producers and consumers must agree on how the data is represented.

Partition Selection

When a producer sends a record to a topic, Kafka needs to determine which partition will store it. If a key is provided, the producer's partitioning logic can use that key to consistently select a partition.

For example:

order-1001 -> Partition 0
order-1002 -> Partition 2
order-1003 -> Partition 1

If the same key is repeatedly used, records for that key are normally routed to the same partition as long as the partitioning configuration remains compatible. This is why choosing a meaningful key, such as orderId or customerId, is an important design decision when ordering matters.

Acknowledgments

A producer also needs to know whether Kafka accepted the record. Kafka provides the acks setting to control how much confirmation the producer requires.

With acks=0, the producer does not wait for an acknowledgment. With acks=1, the leader broker acknowledges the record after accepting it. With acks=all, the leader waits for the required in-sync replicas to acknowledge it, providing stronger durability.

For important business events, acks=all is commonly preferred because losing an OrderCreated event could create serious consistency problems between services.

A .NET Producer

The Confluent.Kafka library is a commonly used Kafka client for .NET. A basic producer can be configured like this:

using Confluent.Kafka;

var config = new ProducerConfig
{
    BootstrapServers = "localhost:9092",
    Acks = Acks.All
};

using var producer = new ProducerBuilder<string, string>(config)
    .Build();

await producer.ProduceAsync(
    "orders",
    new Message<string, string>
    {
        Key = "order-1001",
        Value = "OrderCreated"
    });

Here, BootstrapServers tells the producer how to initially connect to the Kafka cluster. The producer sends the record to the orders topic with order-1001 as its key. Kafka then determines the appropriate partition and stores the record.

The Producer's Journey

Conceptually, the process looks like this:

Order Service
     |
     | Create record
     v
Serialize data
     |
     | Key: order-1001
     v
Select partition
     |
     v
Kafka Broker
     |
     | Acknowledge
     v
Producer

The important point is that the producer does not need to manage the physical location of the topic's partitions itself. It communicates with Kafka, while Kafka's cluster architecture determines where the record belongs and how it is stored.

A Kafka producer therefore does much more than simply "send a message." It creates records, serializes data, chooses partitions, and controls delivery behavior through configuration such as acknowledgments. Understanding these responsibilities prepares us for the other side of the communication: Kafka consumers, which retrieve and process those records.