Unicode

In 1968, ASCII became a standard in the United States. The principle was to assign a numerical value between 0 and 127 to each character. The letter "a" takes code 97, "Z" takes code 90. Simple, effective... but terribly limited. It was impossible to write "naïve" or "café" correctly with this system designed for English.

During the 1980s, personal computing on 8-bit machines took off. The possible values extended up to 255. Manufacturers and publishers rushed to fill these 128 additional positions with accented characters. Each came up with their own solution. Commodore placed French accented letters in its own way, IBM too with its code page 437. A file created on one machine was unreadable on another. What looked like French on an Amstrad CPC turned into hieroglyphics on a PC compatible.

Conventions gradually emerged. The International Organization for Standardization standardized certain encodings, others gained dominance through commercial use. But the fundamental problem remained: 255 characters was laughable compared to actual needs. It was impossible to integrate French accents and the Cyrillic alphabet simultaneously. Users juggled between encodings depending on the language of the moment. KOI8 for typing in Russian, Latin1 for French. Writing a text quoting Tolstoy in the original version was a technical feat.

In the late 1980s, a few visionaries embarked on an ambitious project: creating a system capable of representing all characters from all human languages. Unicode was born from this ambition, with the initial idea of moving to 16-bit characters. Instead of 255 positions, 65,536 slots became available. Enough room for Chinese, Arabic, Hebrew, Cyrillic, and Latin alphabets within the same system.

ISO worked in parallel on its 10646 standard. Two projects, one objective. Fortunately, the teams eventually merged their efforts with Unicode version 1.1. The 65,536 positions proved insufficient, and the modern specification extends to 1,114,111 possible characters (0x10FFFF in hexadecimal notation).

Unicode goes far beyond a simple character catalog. The standard defines complex properties and usage rules. The responsible consortium publishes a manual of over 1,000 pages. Added to this are 14 technical annexes, 7 standards, 6 technical reports, and 4 stabilized reports. Hyphenation, bidirectional rendering, vertical layout: everything is covered. The scope of this documentation reveals the true complexity of multilingual text processing.

But in practice, how do you store these characters in memory? The brute force solution consists of reserving 32 bits per character. But it quadruples disk space and bandwidth compared to ASCII. Not to mention portability problems between processors that organize bytes differently.

UTF-8 provides the elegant answer. This format uses a variable number of bytes depending on the character. A single byte for ASCII (values below 128), two bytes for values up to 2,047, three or four bytes beyond. ASCII compatibility preserved, no troublesome null bytes, possibility of resynchronization in case of corruption: the advantages pile up.

UTF-16 also exists but remains less popular. It encodes characters on two or four bytes with a special mechanism called "surrogate pairs" for characters outside the basic multilingual plane. A byte order marker indicates the organization of data in memory.

Standard updates sometimes create turbulence. Amendment 5 to ISO 10646 moved and extended the Korean Hangul block, invalidating existing data. This episode, dubbed "the Korean mess", pushed standardization committees to commit against this type of incompatible modification.

Unicode manages sophisticated mechanisms. Characters combine, like an accent modifying its base letter. The standard defines bidirectional rendering rules for mixing Arabic texts (right to left) and Western texts (left to right) in the same document.

The adoption of Unicode transformed computing. Operating systems, Internet protocols, file formats: everything shifted toward this unified representation. Software internationalization became greatly facilitated. Exchanging a multilingual document finally became natural.

The Unicode Consortium continues its enrichment work. New characters to cover historical scripts, support for emerging needs: the standard constantly evolves. The arrival of emojis illustrates this capacity to adapt to new uses of digital communication. Each addition requires meticulous care to preserve overall coherence.

Unicode marks a breakthrough in the history of textual computing. This standard establishes a solid technical foundation for multilingual processing. Its universal adoption proves that it is possible to create complex international standards that effectively serve all users on the planet.