Rust Reference ยท Strings and Text
Rust from_utf8 and from_utf8_lossy
Reference for Rust str::from_utf8, String::from_utf8, from_utf8_lossy and Utf8Error: converting bytes to text, error positions and when copying happens.
A str or String must always hold valid UTF-8, so turning arbitrary bytes into text needs a check. std::str::from_utf8 validates a &[u8] and gives you a &str that borrows it, without copying. String::from_utf8 validates a Vec<u8> and takes ownership, again without copying. Both return a Result, with an error that tells you where the first invalid byte is.
When the data may be damaged and you just want something printable, String::from_utf8_lossy replaces each invalid sequence with the replacement character U+FFFD and never fails.
let bytes = [104, 105]; assert_eq!(std::str::from_utf8(&bytes), Ok("hi"));
Try it
Press Run (or Ctrl + Enter) to see the result here.
Program input ยท what you'd type while the program runs
Type the answers before you press Run, one per line. Each time the program reads a line (with io::stdin().read_line()), it gets the next line from this box.
Press Run (or Ctrl + Enter) to see the result here.
Program input ยท what you'd type while the program runs
Type the answers before you press Run, one per line. Each time the program reads a line (with io::stdin().read_line()), it gets the next line from this box.
Syntax
fn std::str::from_utf8(v: &[u8]) -> Result<&str, Utf8Error> fn String::from_utf8(vec: Vec<u8>) -> Result<String, FromUtf8Error> fn String::from_utf8_lossy(v: &[u8]) -> Cow<'_, str> unsafe fn std::str::from_utf8_unchecked(v: &[u8]) -> &str
| Name | Description |
|---|---|
v, vec |
The bytes to interpret as UTF-8. |
Returns: from_utf8 gives Ok(&str) or Err(Utf8Error). String::from_utf8 gives Ok(String) or Err(FromUtf8Error), which holds the original Vec (into_bytes()) and the utf8_error(). from_utf8_lossy returns a Cow<str>: borrowed when the input was valid, owned when replacements were needed.
Errors / panics / edge cases: Utf8Error::valid_up_to() is the length of the valid prefix, and error_len() is Some(n) for an invalid sequence of n bytes or None when the input ended in the middle of a sequence (more bytes might fix it). Nothing here panics. from_utf8_unchecked is unsafe: invalid bytes cause undefined behavior.
Usage notes
- Validation is linear in the length of the input and fast for ASCII. There is no copy for the borrowed and the
Vecversions. - Overlong encodings, surrogates and code points above
U+10FFFFare invalid UTF-8 and are rejected. - Lossy replacement: each maximal invalid subsequence becomes one
U+FFFD("replacement character"), so a damaged 4-byte sequence may become one or several markers. - Streaming data (a network read that stops mid-character) needs care: when
error_len()isNone, keep the leftover bytes and prepend them to the next chunk instead of treating it as corrupt. - Other encodings: Latin-1 bytes are not UTF-8. Convert with
bytes.iter().map(|&b| b as char).collect::<String>()for Latin-1 only; for other legacy encodings use a crate. - Reading a file as text:
fs::read_to_stringfails withInvalidDataon invalid UTF-8; read bytes withfs::readand usefrom_utf8_lossywhen you want to tolerate it. CStr::to_strandOsStr::to_strdo the same validation for C and OS strings.- Use
from_utf8_uncheckedonly when validity is guaranteed by construction, and keepunsafeblocks tiny.