Character Encoding: Why Text Arrives Broken

Computers store bytes. Text is an interpretation of bytes, and the rules for that interpretation are the encoding. When the encoding used to write differs from the one used to read, you get é where é should be — the failure is called mojibake, and it is always the same cause.

Unicode and UTF-8 are different things

Unicode assigns a number — a code point — to every character: U+0041 is A, U+00E9 is é, U+1F600 is a grinning face. It says nothing about bytes.

UTF-8 is one way of writing those numbers as bytes. ASCII characters take one byte, most European characters two, most CJK three, emoji four. Its decisive property is that valid ASCII is valid UTF-8 with identical bytes, which is why it won.

UTF-16 uses two or four bytes and is what JavaScript, Java and Windows use internally. UTF-32 uses four always and is used almost nowhere for storage.

Reading mojibake backwards

The pattern tells you what happened. é for é means UTF-8 bytes were read as Latin-1: é is C3 A9 in UTF-8, and Latin-1 renders those two bytes as two characters. ’ where an apostrophe should be is the same thing applied to a smart quote.

A ? or � instead means the byte could not be represented at all — usually a conversion to a narrower encoding, and unlike mojibake it is not reversible. The information is gone.

Length is three different questions

Take the emoji sequence for a family, or a flag, or an accented letter written as base plus combining mark. "How long is this string" has several correct answers:

• Bytes — what storage and network limits care about.
• Code points — what most languages' length function returns, except in JavaScript and Java where it returns UTF-16 code units.
• Grapheme clusters — what a person would call characters, and what a cursor moves over.

A single emoji can be one grapheme, several code points and a dozen bytes. Truncating a string to "20 characters" by slicing bytes or code units is how you cut a character in half and produce a replacement glyph.

Normalisation

é can be written as one code point (U+00E9) or as e followed by a combining acute (U+0065 U+0301). They look identical and are not equal under a byte comparison.

This breaks deduplication, login by username, and file lookup — macOS normalises filenames one way, Linux does not normalise at all. Normalise to NFC at the point data enters your system and compare after that, not before.

The BOM

A byte-order mark is EF BB BF at the start of a UTF-8 file. It is unnecessary — UTF-8 has no byte-order ambiguity — and Windows tools add it anyway.

The damage is specific: a BOM before <?php produces output before headers are sent, a BOM in a CSV makes the first column header not match its name, and a BOM in a JSON file makes strict parsers reject it. When a file "looks identical" but fails, check the first three bytes.

Getting it right

• UTF-8 everywhere, declared explicitly: <meta charset="utf-8">, Content-Type: text/html; charset=utf-8.
• In MySQL use utf8mb4. The encoding called utf8 is a three-byte subset that cannot store emoji or some CJK characters.
• Decode at the boundary, work in the language's string type, encode on the way out.
• Never guess an encoding from content. Get it from the protocol, the declaration, or a documented convention.

Inspect characters → · HTML entities → · Back to all articles