Skip to content
Tutorials Center
My learning

Topics

Rust Reference ยท Strings and Text

Rust from_utf8 and from_utf8_lossy

Reference for Rust str::from_utf8, String::from_utf8, from_utf8_lossy and Utf8Error: converting bytes to text, error positions and when copying happens.

A str or String must always hold valid UTF-8, so turning arbitrary bytes into text needs a check. std::str::from_utf8 validates a &[u8] and gives you a &str that borrows it, without copying. String::from_utf8 validates a Vec<u8> and takes ownership, again without copying. Both return a Result, with an error that tells you where the first invalid byte is.

When the data may be damaged and you just want something printable, String::from_utf8_lossy replaces each invalid sequence with the replacement character U+FFFD and never fails.

Rust
let bytes = [104, 105];
assert_eq!(std::str::from_utf8(&bytes), Ok("hi"));

Try it

Rust Validating bytes

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with io::stdin().read_line()), it gets the next line from this box.

Rust Lossy conversion

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with io::stdin().read_line()), it gets the next line from this box.

Syntax

Rust
fn std::str::from_utf8(v: &[u8]) -> Result<&str, Utf8Error>
fn String::from_utf8(vec: Vec<u8>) -> Result<String, FromUtf8Error>
fn String::from_utf8_lossy(v: &[u8]) -> Cow<'_, str>
unsafe fn std::str::from_utf8_unchecked(v: &[u8]) -> &str
Name Description
v, vec The bytes to interpret as UTF-8.

Returns: from_utf8 gives Ok(&str) or Err(Utf8Error). String::from_utf8 gives Ok(String) or Err(FromUtf8Error), which holds the original Vec (into_bytes()) and the utf8_error(). from_utf8_lossy returns a Cow<str>: borrowed when the input was valid, owned when replacements were needed.

Errors / panics / edge cases: Utf8Error::valid_up_to() is the length of the valid prefix, and error_len() is Some(n) for an invalid sequence of n bytes or None when the input ended in the middle of a sequence (more bytes might fix it). Nothing here panics. from_utf8_unchecked is unsafe: invalid bytes cause undefined behavior.

Usage notes

  • Validation is linear in the length of the input and fast for ASCII. There is no copy for the borrowed and the Vec versions.
  • Overlong encodings, surrogates and code points above U+10FFFF are invalid UTF-8 and are rejected.
  • Lossy replacement: each maximal invalid subsequence becomes one U+FFFD ("replacement character"), so a damaged 4-byte sequence may become one or several markers.
  • Streaming data (a network read that stops mid-character) needs care: when error_len() is None, keep the leftover bytes and prepend them to the next chunk instead of treating it as corrupt.
  • Other encodings: Latin-1 bytes are not UTF-8. Convert with bytes.iter().map(|&b| b as char).collect::<String>() for Latin-1 only; for other legacy encodings use a crate.
  • Reading a file as text: fs::read_to_string fails with InvalidData on invalid UTF-8; read bytes with fs::read and use from_utf8_lossy when you want to tolerate it.
  • CStr::to_str and OsStr::to_str do the same validation for C and OS strings.
  • Use from_utf8_unchecked only when validity is guaranteed by construction, and keep unsafe blocks tiny.