Elasticsearch

The story of Elasticsearch begins as a love story. In 2004, Shay Banon, a young developer, moved to London to support his wife, who dreamed of becoming a chef. Without employment, he decided to put this time to good use by exploring emerging technologies. His project: to create an application where his wife could store and retrieve all the culinary knowledge acquired during her training.

At the heart of this application, Banon envisioned a simple search box. One would only need to enter a concept, a thought, or an ingredient to immediately explore gastronomic possibilities. This vision led him to Apache Lucene, the reference library for search in Java. His immersion in Lucene gave birth to Compass, a project aimed at simplifying the use of Lucene in conventional Java applications. Conceptually, Compass presented itself as “Hibernate for Lucene”, facilitating the mapping of the domain model to the search index.

The project captivated Banon, who ended up dedicating more time to it than to the initial culinary application. A few months after its creation, he open-sourced Compass. The success was immediate. Developers appreciated the simplicity with which they could now index their domain models in Lucene and perform searches. This freedom gave them access to unexpected and innovative uses.

In 2010, drawing on this experience and user feedback, Banon launched Elasticsearch. This new creation maintained Compass’s philosophy of simplicity while addressing modern needs for distribution and scalability. He left his position at Gigaspaces to devote himself entirely to Elasticsearch development for nearly two years.

The open-source community embraced the project. Developers enriched the features, created an administration console, and developed graphical analysis tools. These collective contributions generated considerable momentum. In 2012, Elasticsearch attracted the attention of chief technology officers from numerous companies who praised its qualities. With 200,000 monthly downloads, an unprecedented figure for enterprise software, Elasticsearch established itself as an essential solution.

The founding team formed around Banon with Steven Schuurman, Uri Boness, and Simon Willnauer, all recognized experts in the search field. Together, they created a company to support the project’s development. The company attracted the best talent from the Lucene community and became the preferred destination for open source in the search domain.

Elasticsearch’s architecture is based on an innovative approach. The system uses JSON and a RESTful interface, making it accessible from any programming language. Its distributed nature gives it the capacity to process large volumes of data by automatically distributing the load across multiple nodes. Schema flexibility allows the storage of complex documents without prior structure definition.

Over the years, the ecosystem expanded. Logstash came to facilitate data ingestion, while Kibana offered advanced visualization capabilities. These tools form the ELK stack (Elasticsearch, Logstash, Kibana), which became a reference for log analysis and infrastructure monitoring. In the 2020s, lighter alternatives such as Fluentd or FluentBit often replaced Logstash for data ingestion.

Elasticsearch use cases multiplied: website search, user interaction analysis, anomaly detection in security systems. Organizations like GitHub, Wikimedia, Foursquare, and CERN adopted it to manage their search and data analysis needs.

In 2018, the company went public, confirming the project’s maturity. The following year marked a significant change. Elastic, the company behind Elasticsearch, modified its license. In January 2021, it abandoned the Apache 2.0 license for a dual proprietary license, including the Server Side Public License (SSPL). This decision prompted a reaction from Amazon Web Services, which created a fork of the project named OpenSearch, maintaining the open-source tradition under the Apache 2.0 license.

Elasticsearch’s impact on the software industry extends beyond its initial objective as a search engine. The technology has redefined how organizations manage and explore their data. Its distributed architecture and API-oriented approach have influenced the design of numerous modern systems. The ease of use, combined with powerful features, gives developers the ability to integrate sophisticated search capabilities without deep expertise in information retrieval. Elasticsearch’s widespread adoption illustrates the growing importance of real-time data analysis in contemporary information systems.

A personal project sometimes evolves into a major infrastructure, used by thousands of organizations around the world.