Skip to content
Tutorials Center
My learning

Topics

Lesson 118 of 201 · Input and Output

Character Sets and Encoding

Understand bytes vs characters in Java - Unicode, UTF-8 and UTF-16, StandardCharsets, getBytes and new String, mojibake, BOMs, malformed input and the UTF-8 default.

Computers store and send bytes, but people read characters. Every time text leaves your program (into a file, a socket, a database or an HTTP response) it has to be turned into bytes, and every time it comes back the bytes have to be turned into characters again. The rule for that conversion is called a character encoding, or in Java's vocabulary a charset. When the writer and the reader use different charsets, café turns into café, names in Greek become question marks, and an emoji becomes four strange symbols. This garbled text even has a name: mojibake, Japanese for "character transformation".

This lesson explains how Unicode, UTF-8 and UTF-16 relate to Java's char and String, how to convert between text and bytes correctly, what mojibake looks like and how to diagnose it, how to make decoding strict instead of silently lossy, and what changed when Java 18 made UTF-8 the default everywhere.

Characters, code points and bytes

Unicode is a catalog that gives every character in every writing system a number called a code point, written like U+0041 for A, U+00E9 for é, U+20AC for € and U+1F600 for 😀. Unicode has room for over a million code points; about 150,000 are assigned.

Unicode says which number a character has. An encoding says how that number is stored as bytes. The important ones:

Encoding Bytes per character Notes
UTF-8 1 to 4 ASCII characters take 1 byte; the standard for files, the web and APIs
UTF-16 2 or 4 How Java stores Strings in memory, conceptually; byte order matters (BE/LE)
ISO-8859-1 (Latin-1) always 1 Only 256 characters: Western European letters, no €
windows-1252 always 1 Latin-1 plus €, curly quotes and a few others; common in old Windows files
US-ASCII always 1 Only 128 characters: English letters, digits and punctuation

Java's char is a 16-bit UTF-16 code unit, not a character. Most characters fit in one char, but code points above U+FFFF, including most emoji, need two chars called a surrogate pair. That's why "😀".length() is 2. (Characters and Booleans introduced this.)

Java The same text in different encodings

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Read the output row by row:

  • A is the single byte 41 in every encoding except UTF-16, which uses two bytes for everything in the basic range.
  • In UTF-8, é and ж take 2 bytes, € and 中 take 3, and 😀 takes 4. Non-ASCII bytes always have the high bit set (c3, e2, f0...), so they can never be confused with ASCII.
  • ISO-8859-1 and US-ASCII can't represent most of these characters. getBytes doesn't complain; it silently writes 3f, the code for ?. That's data loss you won't notice until someone complains about question marks.
  • The emoji is 2 chars, 4 bytes in UTF-16 (the surrogate pair d8 3d de 00) and 4 in UTF-8.

Converting between text and bytes

Two methods do almost all conversions, and both should always get an explicit charset:

Java
byte[] bytes = text.getBytes(StandardCharsets.UTF_8);      // encode: String -> bytes
String text  = new String(bytes, StandardCharsets.UTF_8);   // decode: bytes -> String

StandardCharsets has constants for the six charsets every JVM must support: UTF_8, UTF_16, UTF_16BE, UTF_16LE, ISO_8859_1 and US_ASCII. Others, like windows-1252 or Shift_JIS, are looked up by name with Charset.forName("windows-1252"), which throws UnsupportedCharsetException if the JVM doesn't have it. Prefer the constants: they can't be misspelled and don't throw.

The same rule applies to every API that crosses the text/bytes boundary: InputStreamReader, OutputStreamWriter, Files.newBufferedReader, Files.readString, Scanner, PrintStream, URLEncoder, ByteArrayOutputStream.toString. Each has an overload that takes a Charset. The streams themselves are explained in Byte and Character Streams.

Mojibake: decoding with the wrong charset

Mojibake happens when bytes are written in one charset and read in another. Because UTF-8 and the single-byte charsets agree on ASCII, English text survives and only the accented letters, symbols and emoji turn into garbage, which makes the bug easy to miss in testing:

Java Creating and repairing mojibake

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Learning to recognize the patterns makes diagnosis fast:

  • é, €, “: UTF-8 bytes decoded as windows-1252 or Latin-1. Each multi-byte character becomes 2 or 3 odd symbols. This is the most common kind on the web.
  • � (U+FFFD, the replacement character): bytes that aren't valid in the decoding charset, for example Latin-1 bytes read as UTF-8, or anything non-ASCII read as ASCII. The original bytes are gone; you can't repair it from the string.
  • ?: a character the encoding charset doesn't contain, replaced while writing. Also unrecoverable.

The repair trick works only for the first pattern, and only if every byte survived. Look at the output: the repair is perfect except for the closing quote ”, which came back as �? and makes equals print false. In UTF-8, ” is e2 80 9d, and 9d is one of the few byte values that windows-1252 leaves undefined, so it was decoded to � and the byte was gone before the repair began. Mojibake that passed through a single-byte charset is often almost repairable, which is worse than not at all. The real fix is always to find the place where bytes were decoded without the right charset.

Strict decoding: fail instead of guessing

new String(bytes, charset) and getBytes(charset) never throw: they replace bad input with � or ?. That's convenient for display, but for data import it hides corruption. CharsetDecoder and CharsetEncoder can be told to report errors instead:

Java Detecting invalid UTF-8 with CodingErrorAction.REPORT

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

This is why Files.readString sometimes fails on files that a text editor opens fine: the file isn't UTF-8, and Files refuses to guess. The fix is to pass the file's real charset, as the example does on the last attempt. (Files.newBufferedReader behaves the same way, while Files.lines throws UncheckedIOException wrapping the same error.) There's no reliable way to detect an unknown file's charset automatically; libraries that try can only guess. Agree on UTF-8 with whoever produces your files.

Byte order marks

A byte order mark (BOM) is the invisible character U+FEFF placed at the very start of a file to signal the encoding and, for UTF-16, the byte order. In UTF-8 it's the three bytes EF BB BF. Some Windows tools (notably older Notepad and Excel's "CSV UTF-8" export) add it, and Java's UTF-8 decoder does not remove it. The result is a string whose first character is invisible but real, so firstLine.equals("id,name") fails mysteriously:

Java The invisible BOM at the start of a file

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

When reading files that might come from such tools, strip a leading \uFEFF as shown. When writing UTF-8, don't add a BOM unless a specific consumer (like some versions of Excel) requires it. The UTF-16 lines also show what "byte order" means: big-endian (BE) writes 00 48 for H, little-endian (LE) writes 48 00, and plain UTF_16 adds the BOM fe ff so the reader knows which one it is.

UTF-8 by default since Java 18

Every JVM has a default charset, used by APIs when you don't pass one: new String(bytes), getBytes(), new FileReader(file), new InputStreamReader(in), new PrintStream(out) and others. Until Java 17, the default came from the operating system: UTF-8 on Linux and macOS, but usually windows-1252 on Western Windows machines and other code pages elsewhere. Code that worked on a developer's Mac produced mojibake on a Windows server.

JEP 400 (Java 18) made UTF-8 the default on every platform. You can check what your JVM uses:

Java Inspecting the default charsets

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

What these properties mean:

  • defaultCharset() / file.encoding: what the Java APIs use when no charset is given. UTF-8 since Java 18. Running with -Dfile.encoding=COMPAT restores the old OS-dependent behavior for legacy applications.
  • native.encoding (Java 17): the operating system's own charset, for the rare case you need to read files that other local programs wrote in it. Our server runs with a minimal Linux locale, so it reports ANSI_X3.4-1968, the official name of plain ASCII; before Java 18, that would have been the default charset, and every é written without an explicit charset would have become ?.
  • stdout.encoding / stderr.encoding (Java 19): what System.out and System.err use. These still follow the console, which is why a Windows terminal may show ? for characters Java printed correctly. Running chcp 65001 in the Windows console, or starting Java with -Dstdout.encoding=UTF-8, switches it to UTF-8.

Even with the new default, write the charset explicitly in code that reads or writes external data. It documents intent, keeps the code correct on Java 17 and older, and protects against COMPAT mode. Files methods have always used UTF-8, regardless of the default.

Unicode normalization

One last trap: Unicode can represent some characters in more than one way. é can be the single code point U+00E9, or e followed by the combining accent U+0301. Both look identical on screen, but they're different strings to equals, hashCode and your database. macOS file names and text pasted from some sources often use the decomposed form. java.text.Normalizer converts both to a standard form:

Java
String composed = "caf\u00e9";    // é as one code point
String decomposed = "cafe\u0301"; // e + combining acute accent
composed.equals(decomposed);         // false: lengths are 4 and 5
Normalizer.normalize(decomposed, Normalizer.Form.NFC).equals(composed);   // true

Normalize user input (usually to NFC) before comparing, hashing or storing it, especially for user names, search terms and file names.

Encoding checklist

  • Use UTF-8 for everything you control: source files, data files, databases, HTTP (Content-Type: text/html; charset=utf-8).
  • Pass StandardCharsets.UTF_8 explicitly whenever you convert between bytes and text.
  • Use character streams (Reader/Writer) for text instead of decoding chunks of bytes yourself; a chunk boundary can split a multi-byte character in half, and decoding each chunk separately corrupts it.
  • Use strict decoding (Files.readString, or a decoder with REPORT) when importing data, so bad input fails loudly.
  • Strip a leading BOM from files produced by Windows tools; normalize to NFC before comparing user input.
  • When you see mojibake, find the step where bytes were decoded with the wrong charset; don't patch the symptom.

Check your understanding

Quick check · 5 questions

of 5 answered You scored / 5 · perfect!

  1. 1. How many bytes does "€".getBytes(StandardCharsets.UTF_8) return?

    The euro sign is U+20AC, which UTF-8 encodes in three bytes, e2 82 ac.
  2. 2. A web page shows "Schön" instead of "Schön". What most likely happened?

    The two-byte UTF-8 sequence for ö (c3 b6) was read as two single-byte characters, Ã and ¶. It's the classic UTF-8-read-as-Latin pattern.
  3. 3. What does "naïve".getBytes(StandardCharsets.US_ASCII) do with the ï?

    getBytes never throws for unmappable characters; it substitutes the charset's replacement byte, which is ? (0x3f) for US-ASCII. Use a CharsetEncoder with REPORT to detect this.
  4. 4. Files.readString(path) throws MalformedInputException, but the file opens fine in a text editor. Why?

    Files methods default to UTF-8 and report invalid bytes instead of guessing. Pass the file's real charset, for example Files.readString(path, Charset.forName("windows-1252")).
  5. 5. What changed in Java 18 with JEP 400?

    APIs that use the default charset now use UTF-8 everywhere, not the OS code page. Console output still follows stdout.encoding, which depends on the terminal.

Summary

  • Unicode assigns code points to characters; an encoding (charset) turns them into bytes. UTF-8 uses 1–4 bytes per character and is the standard for data; Java's char is a UTF-16 code unit.
  • Convert with text.getBytes(charset) and new String(bytes, charset), always passing a charset, preferably from StandardCharsets.
  • Mojibake comes from decoding with the wrong charset: é means UTF-8 read as Latin-1/windows-1252; � and ? mean data was already lost.
  • getBytes and new String silently replace bad data; CharsetDecoder/Encoder with CodingErrorAction.REPORT, and the Files methods, fail loudly instead.
  • Strip a UTF-8 BOM (\uFEFF) from files written by some Windows tools; normalize text with Normalizer (NFC) before comparing.
  • Since Java 18 the default charset is UTF-8 everywhere; console output follows stdout.encoding. Still name the charset in code.

Next: Properties Files and Configuration, where you'll load settings from files, system properties and environment variables.