# Strings and UTF-8

**URL:** <https://internals.rust-lang.org/t/strings-and-utf-8/9057>\
**Category:** language design\
**Created:** [December 18, 2018, 10:11pm UTC](https://internals.rust-lang.org/t/strings-and-utf-8/9057 "2018-12-18T22:11:14Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![Nokel81](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/nokel81/32/3966_2.png) [@Nokel81](https://internals.rust-lang.org/u/Nokel81)\
**Post date:** [December 18, 2018, 10:11pm UTC](https://internals.rust-lang.org/t/strings-and-utf-8/9057/1 "2018-12-18T22:11:14Z")

</div>

I was wondering why the string documentation specifically mentions that they are UTF-8 encoded. Is there a specific reason why an encoding method was locked down?

---

<div class="post-metadata">

**Author:** ![mcy](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/mcy/32/6512_2.png) [@mcy](https://internals.rust-lang.org/u/mcy)\
**Post date:** [December 18, 2018, 10:17pm UTC](https://internals.rust-lang.org/t/strings-and-utf-8/9057/2 "2018-12-18T22:17:18Z")

</div>

UTF-8 is layout-compatible with ASCII (as opposed to, say, UTF-16 in Java and JavaScript) and (to my knowledge) the single most common encoding out there. There’s nothing particularly special about the string types; if you really need to handle something weird like latin-1, I’m sure someone’s sorted out a crate for it.

---

<div class="post-metadata">

**Author:** ![Tom-Phinney](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/tom-phinney/32/3299_2.png) [@Tom-Phinney](https://internals.rust-lang.org/u/Tom-Phinney)\
**Post date:** [December 18, 2018, 10:26pm UTC](https://internals.rust-lang.org/t/strings-and-utf-8/9057/3 "2018-12-18T22:26:28Z")

</div>

> [@Nokel81](#):
>
> Is there a specific reason why an encoding method was locked down?

The Rust ecosystem is built around UTF-8. Any other default encoding would create compatibility problems.

---

<div class="post-metadata">

**Author:** ![notriddle](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/notriddle/32/14082_2.png) [@notriddle](https://internals.rust-lang.org/u/notriddle)\
**Post date:** [December 18, 2018, 10:42pm UTC](https://internals.rust-lang.org/t/strings-and-utf-8/9057/4 "2018-12-18T22:42:02Z")

</div>

It means you can convert a UTF-8 encoded byte-blob to a `str` without having to copy it, guaranteed.

---

<div class="post-metadata">

**Author:** ![H2CO3](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/h2co3/32/2849_2.png) [@H2CO3](https://internals.rust-lang.org/u/H2CO3)\
**Post date:** [December 18, 2018, 11:27pm UTC](https://internals.rust-lang.org/t/strings-and-utf-8/9057/5 "2018-12-18T23:27:02Z")

</div>

> [@Nokel81](#):
>
> Is there a specific reason

UTF-8 is basically the best option we have for string encoding today. There are **many** reasons, so I won't repeat them here, because there's a great piece of writing about all the arguments as to why everyone should use UTF-8 (by default and wherever possible): [https://utf8everywhere.org/](https://utf8everywhere.org/)

---

<div class="post-metadata">

**Author:** ![burntsushi](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/burntsushi/32/279_2.png) [@burntsushi](https://internals.rust-lang.org/u/burntsushi)\
**Post date:** [December 18, 2018, 11:58pm UTC](https://internals.rust-lang.org/t/strings-and-utf-8/9057/6 "2018-12-18T23:58:02Z")

</div>

See also this document by Mark Davis, which surveys different string models and their trade offs: [https://docs.google.com/document/d/1wuzzMOvKOJw93SWZAqoim1VUl9mloUxE0W6Ki\_G23tw](https://docs.google.com/document/d/1wuzzMOvKOJw93SWZAqoim1VUl9mloUxE0W6Ki_G23tw)

---

<div class="post-metadata">

**Author:** ![scottmcm](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/scottmcm/32/2355_2.png) [@scottmcm](https://internals.rust-lang.org/u/scottmcm)\
**Post date:** [December 19, 2018, 9:14am UTC](https://internals.rust-lang.org/t/strings-and-utf-8/9057/7 "2018-12-19T09:14:04Z")

</div>

> [@Nokel81](#):
>
> Is there a specific reason why an encoding method was locked down?

Because it affects the possible APIs; most importantly which ones can be no-fail and non-reallocating.

---

<div class="post-metadata">

**Author:** ![josh](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/josh/32/5934_2.png) [@josh](https://internals.rust-lang.org/u/josh)\
**Post date:** [December 19, 2018, 6:19pm UTC](https://internals.rust-lang.org/t/strings-and-utf-8/9057/8 "2018-12-19T18:19:25Z")

</div>

> [@Nokel81](#):
>
> I was wondering why the string documentation specifically mentions that they are UTF-8 encoded. Is there a specific reason why an encoding method was locked down?

Without an encoding specified, you don't have a string, you have bytes. If you want bytes, you can always use `[u8]` or `Vec<u8>`. `str` and `String` exist to contain Unicode characters, not just bytes. Of the ways to encode Unicode characters, UTF-8 provides a far more efficient encoding than UCS-4.

---

<div class="post-metadata">

**Author:** ![burntsushi](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/burntsushi/32/279_2.png) [@burntsushi](https://internals.rust-lang.org/u/burntsushi)\
**Post date:** [December 19, 2018, 6:37pm UTC](https://internals.rust-lang.org/t/strings-and-utf-8/9057/9 "2018-12-19T18:37:29Z")

</div>

FWIW, I’ve been working on a new string library that mirrors Rust’s `String`/`&str` types, except instead of “strings are guaranteed to be valid UTF-8,” they are instead “conventionally UTF-8.” The motivation of the library is to address a very common pain point that I’ve come across over the years where I want to treat bytes as primarily UTF-8 without doing an upfront UTF-8 validation check. Most of the APIs remain the same, with the exception that the Unicode replacement codepoint features more prominently in operations that are only defined on Unicode codepoints (e.g., UTF-8 decoding, grapheme clusters and so on). In particular, The `Vec<u8>`/`&[u8]` types are insufficient to address this use case, since they lack a lot of the common operations one associates with strings.

---

<div class="post-metadata">

**Author:** ![josh](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/josh/32/5934_2.png) [@josh](https://internals.rust-lang.org/u/josh)\
**Post date:** [December 19, 2018, 7:05pm UTC](https://internals.rust-lang.org/t/strings-and-utf-8/9057/10 "2018-12-19T19:05:51Z")

</div>

That sounds like a type I’d love to see in the standard library, eventually.

---

<div class="post-metadata">

**Author:** ![mcy](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/mcy/32/6512_2.png) [@mcy](https://internals.rust-lang.org/u/mcy)\
**Post date:** [December 20, 2018, 1:05am UTC](https://internals.rust-lang.org/t/strings-and-utf-8/9057/11 "2018-12-20T01:05:17Z")

</div>

> [@H2CO3](#):
>
> [https://utf8everywhere.org/](https://utf8everywhere.org/)

This is my new favorite thing. Thank you.

* * *

The following is probably insane:

I wonder how bad of an idea it would be to change `String` to be `String<E: Encoding = Utf8>`. I don't believe this would be a breaking change, though it would be absolutely _awful_ to deal with making `str` generic in the encoding, since it's stuck as a magic builtin for obnoxious historical reasons.

I'm going to go ahead and work out the `Encoding` trait, mostly to convince myself this is insane.

```rust
/// Represents an encoding of Unicode.
trait Encoding {
  /// The type representing a code unit for this
  /// encoding. For UTF-8, this is a `u8`.
  type CodeUnit;

  /// Wrap a stream of bytes encoding valid UTF-8
  /// into a stream of whatever this encoding's codepoints are.
  fn from_utf8(stream: impl Iterator<Item = u8>)
    -> impl Iterator<Item = Self::CodeUnit>;

  /// Wrap a stream using this encoding into a stream of UTF-8 bytes.
  fn to_utf8(stream: impl Iterator<Item = Self::CodeUnit>)
    -> impl Iterator<Item = u8>;
  
  /// Wrap a stream of Unicode codepoints into a stream encoding
  /// using this encoding. 
  fn from_codepoints(stream: impl Iterator<Item = char>)
    -> impl Iterator<Item = Self::CodeUnit>;
  
  /// Wrap a stream using this encoding into a stream of Unicode
  /// codepoints.
  fn to_codepoints(stream: impl Iterator<Item = Self::CodeUnit>)
    -> impl Iterator<Item = char>;
  
  /// Validate that this stream is, in fact, a stream using this encoding.
  fn validate(stream: impl Iterator<Item = Self::CodeUnit>) -> bool;
}

struct Utf8;
impl Encoding for Utf8 {
  type CodeUnit = u8;
  // ..
}

```

Ok I think I might have accidentally written down something reasonable. I really really don't actually think `String` should be generic in encoding; making it this easy to reach for alternate encodings is going to just confuse people, instead of driving use towards the One True Encoding...

Edit: apparently I reinvented `std::basic_string` by accident? Probably worth disregarding this whole post.

---

<div class="post-metadata">

**Author:** ![luben](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/luben/32/3541_2.png) [@luben](https://internals.rust-lang.org/u/luben)\
**Post date:** [December 20, 2018, 2:21am UTC](https://internals.rust-lang.org/t/strings-and-utf-8/9057/12 "2018-12-20T02:21:55Z")

</div>

@burntsushi, do you intend to make the storage pluggable à la `string` crate? I think there is definetely a need for more flexible strings in both axes of correctness guarantees and storage.

---

<div class="post-metadata">

**Author:** ![burntsushi](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/burntsushi/32/279_2.png) [@burntsushi](https://internals.rust-lang.org/u/burntsushi)\
**Post date:** [December 20, 2018, 2:44am UTC](https://internals.rust-lang.org/t/strings-and-utf-8/9057/13 "2018-12-20T02:44:25Z")

</div>

No. Its internal representation is `Vec<u8>`/`&[u8]`.

---

<div class="post-metadata">

**Author:** ![hsivonen](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/hsivonen/32/4080_2.png) [@hsivonen](https://internals.rust-lang.org/u/hsivonen)\
**Post date:** [December 20, 2018, 3:15pm UTC](https://internals.rust-lang.org/t/strings-and-utf-8/9057/14 "2018-12-20T15:15:59Z")

</div>

> [@burntsushi](#):
>
> In particular, The `Vec<u8>` / `&[u8]` types are insufficient to address this use case, since they lack a lot of the common operations one associates with strings.

Do you have examples now that `make_ascii_lowercase()`, etc., exist?

---

<div class="post-metadata">

**Author:** ![burntsushi](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/burntsushi/32/279_2.png) [@burntsushi](https://internals.rust-lang.org/u/burntsushi)\
**Post date:** [December 20, 2018, 3:38pm UTC](https://internals.rust-lang.org/t/strings-and-utf-8/9057/15 "2018-12-20T15:38:12Z")

</div>

If you take the APIs of `String`/`&str` and subtract the APIs of `Vec<u8>`/`&[u8]`, then I think whatever you have leftover would be the starting set of examples. e.g., `chars` (Unicode aware), `replace`, `to_lowercase` (Unicode aware), `to_uppercase` (Unicode aware), the various split routines (Unicode aware), `lines`, `find`, `rfind`, the various trim routines, and so on. Then there are the various ecosystem Unicode crates, such as detection of various boundaries (lines, grapheme clusters, words, sentences). All of this stuff only works on strings that are guaranteed to be valid UTF-8, but all of these operations can be implemented for strings that are only conventionally UTF-8 by making choices about what to do in the presence of invalid UTF-8.

---

<div class="post-metadata">

**Author:** ![hsivonen](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/hsivonen/32/4080_2.png) [@hsivonen](https://internals.rust-lang.org/u/hsivonen)\
**Post date:** [December 20, 2018, 4:38pm UTC](https://internals.rust-lang.org/t/strings-and-utf-8/9057/16 "2018-12-20T16:38:21Z")

</div>

> [@burntsushi](#):
>
> All of this stuff only works on strings that are guaranteed to be valid UTF-8, but all of these operations can be implemented for strings that are only conventionally UTF-8 by making choices about what to do in the presence of invalid UTF-8.

I think it provides significant ecosystem value that crates can focus on the guaranteed-valid case and don't need to also implement the potentially-invalid case.

That said, `&[u8]` could pretty easily provide an iterator that yields `char`s from potentially-invalid UTF-8 with WHATWG Encoding Standard-compliand U+FFFD generation and splitting to lines based on LF and CRLF.

---

<div class="post-metadata">

**Author:** ![H2CO3](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/h2co3/32/2849_2.png) [@H2CO3](https://internals.rust-lang.org/u/H2CO3)\
**Post date:** [December 20, 2018, 5:17pm UTC](https://internals.rust-lang.org/t/strings-and-utf-8/9057/17 "2018-12-20T17:17:00Z")

</div>

To part 1, you’re welcome, and to part 2, please don’t 😃

That said, there’s absolutely no doubt on my part that we _need_ to handle many other sorts of string encodings, but the reason why subtyping or generics doesn’t feel the right solution is that many encodings are just _so_ much different (from UTF-8 and from each other), they have different typical/idiomatic use cases, wildly varying performance characteristics, etc. that bringing them under the same name sounds like asking for trouble.

This is one of the very rare cases where I think “overgeneralization” could be considered a real problem, and we should instead consider making `UTF16String`, `Latin1String`, etc., most probably in small, pluggable, external (non-std) crates. A trait describing the true/sensible intersection of the sets of common operations couldn’t hurt either (e.g. iteration over Unicode code points or grapheme clusters), but it would indeed be a pain to make `String` retroactively generic.

---

<div class="post-metadata">

**Author:** ![mcy](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/mcy/32/6512_2.png) [@mcy](https://internals.rust-lang.org/u/mcy)\
**Post date:** [December 20, 2018, 5:20pm UTC](https://internals.rust-lang.org/t/strings-and-utf-8/9057/18 "2018-12-20T17:20:24Z")

</div>

> [@H2CO3](#):
>
> please don’t

You and me, too.

---

<div class="post-metadata">

**Author:** ![mcy](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/mcy/32/6512_2.png) [@mcy](https://internals.rust-lang.org/u/mcy)\
**Post date:** [December 20, 2018, 5:27pm UTC](https://internals.rust-lang.org/t/strings-and-utf-8/9057/19 "2018-12-20T17:27:59Z")

</div>

> [@H2CO3](#):
>
> but it would indeed be a pain to make `String` retroactively generic.

I don't think it would be totally insane to do the C++ `std::basic_string` thing (i,i `std::string::EncodedString` for us?) and make `type String = EncodedString<Utf8>`. Since `EncodedString` would consist of a `Vec` of code units, we keep the "`String` is a byte vec" guarantee. What I want to avoid is having to reimplement all of the string methods every time.

---

<div class="post-metadata">

**Author:** ![burntsushi](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/burntsushi/32/279_2.png) [@burntsushi](https://internals.rust-lang.org/u/burntsushi)\
**Post date:** [December 20, 2018, 6:07pm UTC](https://internals.rust-lang.org/t/strings-and-utf-8/9057/20 "2018-12-20T18:07:15Z")

</div>

> [@hsivonen](#):
>
> I think it provides significant ecosystem value that crates can focus on the guaranteed-valid case and don’t need to also implement the potentially-invalid case.

Yes... I agree. My crate is for the case where it is inconvenient or sub-optimal to require valid UTF-8 because the world we live in is not valid UTF-8 100% of the time. (For example, ripgrep, or any other tool that needs or wants to read the contents of files on a system that does not guarantee anything about the encoding of said files.)

[Next page](https://internals.rust-lang.org/t/strings-and-utf-8/9057.md?page=2)