Kafka

In 2008, at LinkedIn, Jay Kreps faced a challenge that no one had really anticipated. The professional social network was generating millions of messages each day, and traditional systems struggled to keep pace. Classic transactional data—profiles, user relationships—remained manageable. But it was primarily the mass of behavioral information that posed problems: profile views, keyword searches, various interactions. These digital traces, essential for understanding member expectations, literally overwhelmed the existing infrastructure.

Kreps then decided to develop a new solution. He partnered in 2010 with Neha Narkhede and Jun Rao to finalize what would become Apache Kafka. The name references writer Franz Kafka, a nod to the software’s primary purpose: efficiently managing message stream writing. The team designed a distributed platform capable of broadcasting events in real time, with an architecture designed to handle considerable volumes.

The idea that makes all the difference lies in a counter-intuitive approach to storage. Where classic solutions keep messages in RAM, Kafka writes them directly to hard disk, sequentially. This method takes advantage of the natural optimization of linear input-output operations on hard drives, delivering impressive throughput without sacrificing reliability.

LinkedIn released Kafka as open source in 2011, under the aegis of the Apache Foundation. Silicon Valley giants took interest: Twitter, Netflix, and Yahoo adopted the technology, soon followed by more traditional companies like Goldman Sachs, Walmart, and Cerner. At LinkedIn, the numbers are staggering: over 800 billion messages processed daily, representing more than 175 terabytes of data flowing flawlessly.

In 2014, the three creators founded Confluent. Their objective: develop the ecosystem around Kafka and offer enterprise solutions based on this technology. The impact was immediate: the adoption rate multiplied sevenfold in the eleven months following the company’s launch.

The architecture rests on simple but effective concepts. Messages are organized into "topics," categories that structure data flows. Each topic divides into partitions, replicated across different servers to guarantee availability and fault tolerance. This organization distributes the load and scales the system horizontally, without theoretical limit.

Development continued with major features. Kafka Connect, available since version 0.9, simplifies integration with other systems through a standardized framework for connectors. Kafka Streams offers a stream processing library directly integrated into applications, without additional infrastructure to deploy.

In 2022, KRaft (Kafka Raft) was released, a native consensus mechanism that replaces the historical dependency on ZooKeeper. The architecture becomes simpler and more reliable. This year also saw the introduction of tiered storage: part of the data can now be offloaded to object storage systems like S3, significantly reducing operating costs.

Kafka has transformed how companies think about their real-time data processing architectures. The model has inspired numerous alternatives: RedPanda offers an implementation compatible with the Kafka protocol, while WarpStream develops a radically different architecture favoring S3 storage.

In 2024, Kafka has established itself as a mature technology. Its adoption continues to grow, driven by the constant increase in real-time processing needs and the rise of the Internet of Things. The community that formed around the project regularly contributes to its improvement, ensuring continuous evolution.

The success stems from a solid technical architecture, a precise response to a real need, and a dynamic community supported by a stable company. What started as an internal project at LinkedIn has become a cornerstone of modern architectures.