Internet Archive
Brewster Kahle created Internet Archive in 1996. He wanted to build a universal digital library that would preserve the history of the Internet. Not an archive locked away in a digital vault, but an open space where anyone could retrieve vanished web pages. The Wayback Machine, launched in October 2001, provides access 24 years later to over one trillion web pages and more than 99 petabytes of data. Staggering.
Kahle established simple rules from the start. Use only standard hardware. Ban commercial software. Keep the architecture simple enough that you don’t need a PhD to maintain it. These pragmatic choices probably explain why the service has been running for over twenty years with fewer than five people managing it. An almost incredible longevity in the web universe.
The infrastructure relies on distributed storage nodes and front-end web servers. Data is distributed across more than 2,500 nodes, representing over 6,000 hard drives. Total capacity approaches one petabyte. Each day, the system processes tens of millions of requests and transfers more than 40 terabytes of data.
Unlike conventional websites, Internet Archive doesn’t use caching. This absence is surprising at first glance. But the designers quickly realized that archive queries are too dispersed in time and space. A traditional cache would serve no purpose. They preferred to directly optimize access to stored data.
The ARC format structures the storage of archived pages. Each file groups approximately 100 megabytes of unrelated pages, along with their headers. The organization makes storage efficient but requires sequential reading to retrieve a specific page. Each file exists on at least two different nodes, to avoid losing everything in case of failure.
To locate data, the system adopts an original method. No centralized index. Queries are sent via UDP to all storage nodes. Each node keeps the list of its files in memory and responds if it has what’s being sought. This distributed approach withstands failures well and simplifies adding or removing nodes.
Statistics show that English overwhelms other languages in accessed content. European languages follow, far behind. More than 82% of human sessions arrive through external links. Wikipedia leads the sites that send traffic to Internet Archive. Most pages accessed no longer exist on the active web. Proof that the archive fulfills its role as memory.
Two types of users share access: humans and robots. Robotic sessions are ten times more numerous but generate as much data as human sessions. Robots never arrive with a referrer in their requests, whereas human users almost always arrive via links.
The web’s growth imposes constant challenges. The index of archived URLs exceeds 2 terabytes. Initially, updating it was problematic. Disks overheated under intense activity. Moving to a better-cooled data center and an incremental update method solved the problem. Moreover, SSD drives open new possibilities. They outperform mechanical disks for random access, frequent in archive usage. These technologies could make cache systems efficient, previously impossible with traditional media.
Internet Archive collaborates with other institutions like Bibliotheca Alexandrina in Egypt or the European Archives. These collaborations serve to replicate data geographically. Availability improves, as does preservation. The service also offers specialized collections: the Million Book Project, the Prelinger Archives. The mission expands beyond web archiving.
In 2024, Google Search facilitates access to the past more than ever. Archived versions of web pages are directly accessible with a simple link to the Internet Archive’s Wayback Machine.
Internet Archive demonstrates that a minimalist approach can result in a sustainable service. Its architecture favors simplicity and robustness over sophistication. A large-scale archiving system is capable of functioning with a small team. This success now inspires other digital archiving projects worldwide.