On 11 January 2011 LinkedIn released Kafka under the Apache 2.0 licence and described it as “a persistent, efficient, distributed message queue”. What sets it apart from conventional brokers is a structural choice: the system keeps no per-message delivery state, but exposes a persistent append-only log from which each consumer reads at its own position.

Context and problem

Kafka was built to track the activity events of a high-traffic site: page views, search queries, ads served. LinkedIn speaks of an enormous daily volume of events, to be collected with low latency and made available to several downstream systems — indexing, analysis, monitoring.

Queue brokers, such as those that speak JMS (Java Message Service) or AMQP (Advanced Message Queuing Protocol), have a different semantics. A message reaches a consumer and then disappears from the queue; the broker tracks every delivery and every acknowledgement. For transactional work that suits well: each message is an action to be performed exactly once. It suits far worse when the same stream of events has to pass through several independent systems, each at its own pace, and when the volume makes it costly to keep, message by message, the state of every recipient.

Architecture

Kafka gathers messages into topics. Each topic is split into partitions, and each partition is an ordered, immutable sequence written to only at the tail. Each message written to a partition gets a sequential number, the offset, which marks its position. Partitions are spread across the cluster nodes, the brokers; it is partitioning that grows throughput horizontally, because producers and consumers work in parallel across different partitions.

A producer publishes by appending to a partition. A consumer reads from an offset and moves forward by asking for the messages that follow: the model is pull-based, that is, the consumer asks the broker for data, not the broker pushing it. It follows that the broker need neither manage nor know the pace of its readers. The state of a read — how far a consumer has got — is just an offset, kept by the consumer itself. Rewinding a consumer or reprocessing a range of messages means reading again from an earlier offset, with no retransmission or duplication on the broker.

Messages are not deleted once read. They stay on disk for a configured retention period; what deletes them is age, not consumption. On the broker, a partition’s log is a directory of files: a sequence of segments written in the same format in which the producer sent them and in which the consumer reads them. With a single format along the whole chain, chunks of the log pass from the operating system’s page cache straight to the network socket through the sendfile call, with no intermediate copies into user space. Kafka thereby offloads to the kernel cache and to the filesystem’s sequential I/O much of the work a conventional broker would do in application memory.

The critical point

What really changes is where consumption state lives. In a queue broker the state lives in the broker: for every message and every recipient it must know whether delivery has happened. That state grows with the number of consumers and with the in-flight messages, and ends up concentrating complexity and cost there. In Kafka, consumption state shrinks to one offset per consumer per partition, and lives outside the write path. The broker does one thing: it appends to the log and serves sequential reads from a given position.

From this comes the property most useful for data pipelines. The same topic can be read together by an indexing process, an analysis process and a monitoring system, each with its own offset, without one reader touching another and without the broker keeping a per-recipient copy. Producer and consumer fully decouple in time: they need not be on at the same moment, because what binds them is the persistent log, not a delivery session.

Implications

Treating the log as a primitive, rather than as an internal detail of a queue, changes the way one looks at event data. A system’s activity stream becomes a durable, re-readable object, not a series of messages to deliver and forget. A consumer added later can start from the beginning of the available retention and rebuild its state from the log, with the event producer knowing nothing about it. For anyone building pipelines that collect and distribute logs and metrics, it is a basis that carries new downstream systems without multiplying the cost on the broker.

The log as a foundation is not a new idea: it is the same primitive beneath database replication and write-ahead logs. Kafka exposes it as a first-class messaging service, with partitioning and configurable retention, instead of keeping it shut inside a single storage system.

Limits

Kafka is a young project: the code released in January 2011, written in Scala, is at version 0.x and should be taken as such. The delivery semantics that the pull-with-offset model makes natural is at-least-once: if a consumer crashes, resuming from a committed offset may reprocess messages already handled, and it falls to the application to handle idempotency. Ordering is guaranteed within a partition, not across the partitions of one topic: anyone who wants a total order must funnel related messages onto a single partition, and give up some parallelism. Finally, filesystem persistence with time-based retention suits high-volume, low-per-unit-value event streams; it does not replace a transactional broker, where each individual message asks for per-message delivery guarantees and point acknowledgements.

https://blog.linkedin.com/2011/01/11/open-source-linkedin-kafka http://sna-projects.com/kafka/ https://www.apache.org/licenses/LICENSE-2.0 https://www.noze.it/en/insights/apache-kafka-open-source/

Cover image: Apache Kafka architecture diagram: three producers at the top, a central Kafka cluster with three topics split into partitions, and… — diagram by Ch.ko123, CC BY 4.0 — https://commons.wikimedia.org/wiki/File:Overview_of_Apache_Kafka.svg