Hadoop

In 2006, Yahoo! faces a major challenge. Its WebMap infrastructure, the technology that builds the web graph to power its search engine, can no longer keep up. The graph contains over 100 billion nodes and 1 000 billion edges. The previous infrastructure, called Dreadnaught, has reached its limit at 800 machines. The architecture needs a complete overhaul to keep pace with the growth of the web. Yahoo! then chooses Apache Hadoop.

The story begins two years earlier. In 2004, researchers at Google publish two papers that will change everything. They describe their file system and the MapReduce architecture. These publications establish the technical foundation: a distributed file system capable of storing immense data volumes, combined with a model for processing them in parallel at large scale.

The first version of Hadoop emerges within the Apache Nutch project, before becoming independent in 2006. Four years later, Google obtains a patent on the MapReduce algorithm, but grants a license to the Apache Software Foundation. This decision paves the way for massive industry adoption.

The initial architecture rests on two pillars. HDFS (Hadoop Distributed File System) manages distributed data storage. MapReduce coordinates the execution of processing tasks. A central service, the JobTracker, oversees all operations, while TaskTrackers execute tasks on each cluster node.

This monolithic architecture quickly reveals its weaknesses. Developers attempt to repurpose MapReduce for uses it was never intended for. Some launch map-only tasks to run web servers or iterative calculations. These workarounds, despite their ingenuity, compromise the system’s stability and efficiency.

The Apache community undertakes a major overhaul. YARN (Yet Another Resource Negotiator) emerges. This new version separates resource management from the programming model. A central ResourceManager allocates cluster resources to applications. Each application has its own ApplicationMaster to coordinate its execution.

This architectural redesign transforms the ecosystem. New frameworks emerge: Apache Tez for executing task graphs, Spark for in-memory processing, Storm for real-time processing, Giraph for graph computing. Users are no longer constrained by MapReduce’s limitations.

Hadoop adoption in the industry reaches considerable scale. Yahoo! deploys YARN across all its production clusters. The system processes approximately 500 000 jobs daily, representing over 230 years of computation. On a 2 500-machine cluster, the transition to YARN doubles average CPU utilization. The company runs twice as many tasks as before.

Performance improves. In 2013, the MapReduce implementation on YARN sets records in the Daytona and Indy GraySort benchmarks. The system sorts 1.42 terabytes of data per minute on a 2 100-node cluster. These results demonstrate its capability to efficiently process massive volumes.

Hadoop’s architecture influences the design of modern distributed systems. Its ability to run on commodity hardware, its fault tolerance, and its simple programming model make it a reference point. The concept of data locality, where computation moves to the data rather than the reverse, becomes a defining principle of distributed computing.

Hadoop’s success inspires the development of numerous competing platforms. Twitter creates Mesos, Facebook designs Corona, Google develops Omega. Each system brings its innovations, but all share Hadoop’s architectural heritage: the separation between resource management and programming models.

Hadoop’s evolution reflects the transformation of data processing needs. From a specialized tool for web indexing, it transforms into a generic platform supporting a multitude of applications. This transformation comes with continuous improvements in performance, flexibility, and ease of use.

By demonstrating the viability of large-scale distributed processing on commodity hardware, Hadoop democratizes massive data analysis. Companies of all sizes can now build high-performance data processing infrastructures without investing in expensive specialized hardware. The collaboration between companies, developers, and researchers within the Apache Foundation has enabled rapid advances and widespread knowledge sharing.