UTF-8000 experiment extends UTF-8 structure to arbitrary code-point sizes

An intentionally unlimited encoding design

A technical project published on September 20 has proposed “UTF-8000,” an experimental variable-length encoding that extends the structural ideas behind UTF-8 to represent arbitrarily large non-negative integers. The design is not a change to the Unicode standard. Instead, it explores how UTF-8's recognizable byte patterns and built-in length signals could be generalized when a code unit needs more bytes than the current format permits.

Standard UTF-8 uses one byte for ASCII and up to four bytes for other Unicode code points. UTF-8000 retains ASCII unchanged and keeps the familiar distinction between a first byte and continuation bytes. In the proposal, bytes beginning with a zero bit are ASCII, while multibyte units begin with a `11` prefix. Continuation bytes retain the `10` prefix used by UTF-8.

Those prefixes allow a decoder to identify the kind of byte it has encountered without reading everything that came before it. The project describes this as self-synchronization: after seeking to an arbitrary position in a file or recovering from damaged input, a decoder can recognize continuation bytes and move forward until it finds a valid starting point.

Start bits spread across multiple bytes

The main departure from UTF-8 is how UTF-8000 records the length of very large code units. UTF-8's length marker fits in its first byte. UTF-8000 permits the marker, expressed as a run of one bits followed by a terminating zero, to continue through additional bytes near the beginning of the unit. The proposal calls the bytes carrying those markers “start bytes,” even when some are also continuation bytes.

For a unit containing n bytes, the design uses n-2 one bits followed by zero to signal the length. When the marker no longer fits in the first byte, it is distributed through available positions in subsequent bytes while their continuation prefixes remain intact. Once the terminating zero is read, the decoder knows exactly how many bytes belong to the unit.

The site uses a 22-byte example to illustrate the mechanism, emphasizing that the size is only a demonstration and not a special case. It says the same rules can continue to larger units, which is why the format is described as unlimited. It also carries forward checks intended to prevent overlong encodings, where a value is represented with more bytes than necessary.

UTF-8000 is best understood as a specification exercise rather than a replacement proposed for everyday text. Unicode already defines the characters modern systems need, and compatibility depends on standardized UTF-8 behavior. The project nevertheless provides a detailed exploration of why UTF-8 is self-synchronizing and prefix-free, and how those properties might be preserved in a deliberately unconstrained encoding.