# Micro RFC: \`String::from\_utf8\_with\` for different handling of invalid UTF8

**URL:** <https://internals.rust-lang.org/t/micro-rfc-string-from-utf8-with-for-different-handling-of-invalid-utf8/18464>\
**Category:** libs\
**Created:** [March 4, 2023, 6:40am UTC](https://internals.rust-lang.org/t/micro-rfc-string-from-utf8-with-for-different-handling-of-invalid-utf8/18464 "2023-03-04T06:40:22Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![tgross35](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/tgross35/32/9550_2.png) [@tgross35](https://internals.rust-lang.org/u/tgross35)\
**Post date:** [March 4, 2023, 6:40am UTC](https://internals.rust-lang.org/t/micro-rfc-string-from-utf8-with-for-different-handling-of-invalid-utf8/18464/1 "2023-03-04T06:40:22Z")

</div>

Currently, there is only one way to handle invalid UTF8: `String::from_utf8_lossy` and `{OsStr, CStr, Path}::to_string_lossy`, which replace any invalid unicode with the replacement character `�`. However, it is sometimes useful to be able to do something else with invalid characters: either hide them completely, or display them in some usable way.

This is mainly useful for C/OS interfaces, as well as WASM (javascript strings can contain invalid UTF16). However, it is also valuable while debugging to be able to see what exactly is contained in a mostly-utf8 byte slice. Something like this has been indirectly asked for before: [utf 8 - Is it possible to decode bytes to UTF-8, converting errors to escape sequences in Rust? - Stack Overflow](https://stackoverflow.com/questions/41455206/is-it-possible-to-decode-bytes-to-utf-8-converting-errors-to-escape-sequences-i)

Python provides a good reference implementation: [codecs — Codec registry and base classes — Python 3.11.2 documentation](https://docs.python.org/3/library/codecs.html#error-handlers).

This proposal would add a new associated function to accompany any `*_lossy` string-related methods:

```rust
String::from_utf8_with<H: EncodingHandler>(v: &[u8], H) -> Cow<'_, str>;
String::from_utf16_with<H: EncodingHandler>(v: &[u8], H) -> Cow<'_, str>;
OsStr::to_string_with<H: EncodingHandler>(&self, H) -> Cow<'_, str>;
CStr::to_string_with<H: EncodingHandler>(&self, H) -> Cow<'_, str>;
Path::to_string_with<H: EncodingHandler>(&self, H) -> Cow<'_, str>;

```

All of which would take an enum variant that describes how to handle invalid utf8 encoding:

```rust
#[non_exhaustive]
enum Handler {
    /// Remove invalid unicode
    Ignore,
    /// Replace invalid unicode with �.
    /// This is the current behavior of `from_utf8_lossy`
    Replace,
    /// Replace invalid unicode with e.g. "\x9f". This is the equivilent to Python's
    /// `backslashreplace`
    HexEscape, // BackslashReplace would also work
    /// Exact result TBD, some form of "\u{dcf0}", or "\udcf0"
    /// This version may not be as useful
    SurrogateEscape,
}

impl EncodingHandler for Handler {}

/// Allow user-implemented function, best signature TBD. The byte
/// slice would be the invalid chunk, and it should return the intended
/// replacement.
impl<'a, F> EncodingHandler for F where F: Fn(&'a [u8]) -> Cow<'a, str> {}

```

Implementation is fairly trivial - it uses the same basis of `from_utf8_lossy` with slight adjustments to what to do with invalid character ranges. Naming is of course nonfinal.

So, what are thoughts about an official RFC to add these methods?

---

<div class="post-metadata">

**Author:** ![semicoleon](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/semicoleon/32/7397_2.png) [@semicoleon](https://internals.rust-lang.org/u/semicoleon)\
**Post date:** [March 4, 2023, 7:37am UTC](https://internals.rust-lang.org/t/micro-rfc-string-from-utf8-with-for-different-handling-of-invalid-utf8/18464/2 "2023-03-04T07:37:14Z")

</div>

Is there a reason not to take a closure that could "map" the invalid characters instead of using an enum?

---

<div class="post-metadata">

**Author:** ![eggyal](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/eggyal/32/7947_2.png) [@eggyal](https://internals.rust-lang.org/u/eggyal)\
**Post date:** [March 4, 2023, 9:41am UTC](https://internals.rust-lang.org/t/micro-rfc-string-from-utf8-with-for-different-handling-of-invalid-utf8/18464/3 "2023-03-04T09:41:14Z")

</div>

> [@tgross35](#):
>
> So, what are thoughts about an official RFC to add these methods?

These days, this should be submitted as [API Change Proposal](https://std-dev-guide.rust-lang.org/feature-lifecycle/api-change-proposals.html) rather than an RFC.

---

<div class="post-metadata">

**Author:** ![burntsushi](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/burntsushi/32/279_2.png) [@burntsushi](https://internals.rust-lang.org/u/burntsushi)\
**Post date:** [March 4, 2023, 11:41am UTC](https://internals.rust-lang.org/t/micro-rfc-string-from-utf8-with-for-different-handling-of-invalid-utf8/18464/4 "2023-03-04T11:41:18Z")

</div>

I would rather see a function like this added to std: [decode\_utf8 in bstr - Rust](https://docs.rs/bstr/latest/bstr/fn.decode_utf8.html)

Then you can pretty easily implement whatever semantics you want.

(I believe it's true you can use `std::str::from_utf8` to actually implement a `decode_utf8` function today, but it's pretty awkward.)

---

<div class="post-metadata">

**Author:** ![the8472](https://avatars.discourse-cdn.com/v4/letter/t/0ea827/32.png) [@the8472](https://internals.rust-lang.org/u/the8472)\
**Post date:** [March 4, 2023, 1:29pm UTC](https://internals.rust-lang.org/t/micro-rfc-string-from-utf8-with-for-different-handling-of-invalid-utf8/18464/5 "2023-03-04T13:29:43Z")

</div>

For `String` we could add something like `valid_prefix()` and `remainder()` to `FromUtf8Error`. This could also be helpful in other contexts, e.g. when parsing something that's partially utf8, partially binary data.

---

<div class="post-metadata">

**Author:** ![burntsushi](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/burntsushi/32/279_2.png) [@burntsushi](https://internals.rust-lang.org/u/burntsushi)\
**Post date:** [March 4, 2023, 1:58pm UTC](https://internals.rust-lang.org/t/micro-rfc-string-from-utf8-with-for-different-handling-of-invalid-utf8/18464/6 "2023-03-04T13:58:37Z")

</div>

I think we get that by virtue of [`Utf8Error`](https://doc.rust-lang.org/std/str/struct.Utf8Error.html), which you can in turn get from [`FromUtf8Error::utf8_error`](https://doc.rust-lang.org/std/string/struct.FromUtf8Error.html#method.utf8_error).

---

<div class="post-metadata">

**Author:** ![semicoleon](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/semicoleon/32/7397_2.png) [@semicoleon](https://internals.rust-lang.org/u/semicoleon)\
**Post date:** [March 4, 2023, 8:14pm UTC](https://internals.rust-lang.org/t/micro-rfc-string-from-utf8-with-for-different-handling-of-invalid-utf8/18464/7 "2023-03-04T20:14:57Z")

</div>

Does the (unstable) [`Utf8Chunks`](https://doc.rust-lang.org/std/str/struct.Utf8Chunks.html) API enable what you want? That appears to be how the lossy conversion is currently implemented.

---

<div class="post-metadata">

**Author:** ![idanarye](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/idanarye/32/3346_2.png) [@idanarye](https://internals.rust-lang.org/u/idanarye)\
**Post date:** [March 5, 2023, 1:37am UTC](https://internals.rust-lang.org/t/micro-rfc-string-from-utf8-with-for-different-handling-of-invalid-utf8/18464/8 "2023-03-05T01:37:16Z")

</div>

Alternatively, maybe a decode function that accepts a closure that deals with the invalid bytes?

---

<div class="post-metadata">

**Author:** ![8573](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/8573/32/4913_2.png) [@8573](https://internals.rust-lang.org/u/8573)\
**Post date:** [March 5, 2023, 6:18am UTC](https://internals.rust-lang.org/t/micro-rfc-string-from-utf8-with-for-different-handling-of-invalid-utf8/18464/9 "2023-03-05T06:18:51Z")

</div>

> [@burntsushi](#):
>
> I would rather see a function like this added to std: [decode\_utf8 in bstr - Rust](https://docs.rs/bstr/latest/bstr/fn.decode_utf8.html)

How much does that design — yielding (at most) one `char` at a time, with iteration controlled by the caller — limit the ability to apply SIMD? (I ask as someone who is not familiar with SIMD but recalls that you are.)

---

<div class="post-metadata">

**Author:** ![burntsushi](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/burntsushi/32/279_2.png) [@burntsushi](https://internals.rust-lang.org/u/burntsushi)\
**Post date:** [March 5, 2023, 12:08pm UTC](https://internals.rust-lang.org/t/micro-rfc-string-from-utf8-with-for-different-handling-of-invalid-utf8/18464/10 "2023-03-05T12:08:10Z")

</div>

I imagine it would inhibit SIMD or pretty much any other throughput related optimizations. The API is meant for convenience when you just want to pluck a codepoint off of the beginning of a byte slice. In that sense, it is quite versatile.

bstr does still use SIMD for UTF-8 decoding in APIs that permit it. (Such as `ByteSlice::is_utf8`.)

---

<div class="post-metadata">

**Author:** ![user16251](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/user16251/32/10186_2.png) [@user16251](https://internals.rust-lang.org/u/user16251)\
**Post date:** [March 5, 2023, 2:47pm UTC](https://internals.rust-lang.org/t/micro-rfc-string-from-utf8-with-for-different-handling-of-invalid-utf8/18464/11 "2023-03-05T14:47:55Z")

</div>

Instead of a non-exhaustive enum for behaviors, could this be captured in a trait and types implementing it?

---

<div class="post-metadata">

**Author:** ![tgross35](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/tgross35/32/9550_2.png) [@tgross35](https://internals.rust-lang.org/u/tgross35)\
**Post date:** [March 6, 2023, 3:17pm UTC](https://internals.rust-lang.org/t/micro-rfc-string-from-utf8-with-for-different-handling-of-invalid-utf8/18464/12 "2023-03-06T15:17:22Z")

</div>

> Is there a reason not to take a closure that could "map" the invalid characters instead of using an enum?

> Alternatively, maybe a decode function that accepts a closure that deals with the invalid bytes?

> Instead of a non-exhaustive enum for behaviors, could this be captured in a trait and types implementing it?

These are all good points - I updated the sample to be trait-based so the user can write their own closure. The reason for providing the enum at all was just to provide some commonly used defaults.

> These days, this should be submitted as [API Change Proposal](https://std-dev-guide.rust-lang.org/feature-lifecycle/api-change-proposals.html) rather than an RFC.

Thanks for the clarification, I suppose it's still good to collect feedback here before actually proposing something.

---

<div class="post-metadata">

**Author:** ![tgross35](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/tgross35/32/9550_2.png) [@tgross35](https://internals.rust-lang.org/u/tgross35)\
**Post date:** [March 6, 2023, 3:18pm UTC](https://internals.rust-lang.org/t/micro-rfc-string-from-utf8-with-for-different-handling-of-invalid-utf8/18464/13 "2023-03-06T15:18:45Z")

</div>

> I would rather see a function like this added to std: [decode\_utf8 in bstr - Rust](https://docs.rs/bstr/latest/bstr/fn.decode_utf8.html)
> 
> Then you can pretty easily implement whatever semantics you want.

That is also not a bad idea in any case, and also a good option if this isn't seen as fit for `std`. I don't think it's a far stretch for `std` though, since surface area is fairly minimal and use cases aren't uncommon (e.g., wanting to display a path without � or wanting to retain some information about what characters are invalid)

> (I believe it's true you can use `std::str::from_utf8` to actually implement a `decode_utf8` function today, but it's pretty awkward.)

That's exactly what I did for our regex wasm thingy, [but it's clunky](https://github.com/tgross35/wasm-regex/blob/89669b353991d65c5d36135f0c5a92ca0d186fb3/src/lib.rs#L121-L168) (there's room for cleanup, but I don't think that much). That's what led me to write this up 🙂

---

<div class="post-metadata">

**Author:** ![burntsushi](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/burntsushi/32/279_2.png) [@burntsushi](https://internals.rust-lang.org/u/burntsushi)\
**Post date:** [March 6, 2023, 3:31pm UTC](https://internals.rust-lang.org/t/micro-rfc-string-from-utf8-with-for-different-handling-of-invalid-utf8/18464/14 "2023-03-06T15:31:26Z")

</div>

> [@tgross35](#):
>
> That is also not a bad idea in any case, and also a good option if this isn't seen as fit for `std`. I don't think it's a far stretch for `std` though, since surface area is fairly minimal and use cases aren't uncommon (e.g., wanting to display a path without � or wanting to retain some information about what characters are invalid)

I think if you're adding traits and what not, then the API surface is not something I personally would consider minimal. (In case you didn't know, I am on `libs-api`.) For something like that, my first response to any such proposal would be to prototype it in a crate first and get people to use it. Because there's no specific reason it needs to live in `std`.

The nice thing about `bstr::decode_utf8` is that it's extremely flexible and basically lets anyone implement whatever they want at the very lowest layer. That's what I would consider minimal. 🙂

---

<div class="post-metadata">

**Author:** ![tgross35](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/tgross35/32/9550_2.png) [@tgross35](https://internals.rust-lang.org/u/tgross35)\
**Post date:** [March 6, 2023, 3:35pm UTC](https://internals.rust-lang.org/t/micro-rfc-string-from-utf8-with-for-different-handling-of-invalid-utf8/18464/15 "2023-03-06T15:35:14Z")

</div>

> Does the (unstable) [`Utf8Chunks`](https://doc.rust-lang.org/std/str/struct.Utf8Chunks.html) API enable what you want? That appears to be how the lossy conversion is currently implemented.

It does allow for the implementation! At least something less clunky than [what I linked in this reply](https://internals.rust-lang.org/t/micro-rfc-string-from-utf8-lossy-using-for-different-handling-of-invalid-utf8/18464/13). There may be a better place for this API within/using `Utf8Chunks` somehow, but adding some default implementations for the most common cases is what I'm hoping for.

---

<div class="post-metadata">

**Author:** ![tgross35](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/tgross35/32/9550_2.png) [@tgross35](https://internals.rust-lang.org/u/tgross35)\
**Post date:** [March 6, 2023, 4:13pm UTC](https://internals.rust-lang.org/t/micro-rfc-string-from-utf8-with-for-different-handling-of-invalid-utf8/18464/16 "2023-03-06T16:13:29Z")

</div>

> I think if you're adding traits and what not, then the API surface is not something I personally would consider minimal. (In case you didn't know, I am on `libs-api`.) For something like that, my first response to any such proposal would be to prototype it in a crate first and get people to use it. Because there's no specific reason it needs to live in `std`.
> 
> The nice thing about `bstr::decode_utf8` is that it's extremely flexible and basically lets anyone implement whatever they want at the very lowest layer. That's what I would consider minimal. 🙂

I suppose I meant moreso that added maintenance area is fairly minimal, since it would consume the current implementation of `from_utf8_lossy`.

Is your thought for `bstr` something like a `decode_utf8_with` that returns `(Result<char, u8>, usize)` or `Result<(char, usize), u8>`, to parallel `decode_utf8` but allow special handling of the invalid byte?

---

<div class="post-metadata">

**Author:** ![burntsushi](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/burntsushi/32/279_2.png) [@burntsushi](https://internals.rust-lang.org/u/burntsushi)\
**Post date:** [March 6, 2023, 4:27pm UTC](https://internals.rust-lang.org/t/micro-rfc-string-from-utf8-with-for-different-handling-of-invalid-utf8/18464/17 "2023-03-06T16:27:30Z")

</div>

> [@tgross35](#):
>
> Is your thought for `bstr` something like a `decode_utf8_with` that returns `(Result<char, u8>, usize)` or `Result<(char, usize), u8>`, to parallel `decode_utf8` but allow special handling of the invalid byte?

I'm not sure what you mean exactly. `bstr::decode_utf8`, as it currently exists, should be able to handle all of the use cases you've outlined. I don't know why a `decode_utf8_with` would be needed.

---

<div class="post-metadata">

**Author:** ![tgross35](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/tgross35/32/9550_2.png) [@tgross35](https://internals.rust-lang.org/u/tgross35)\
**Post date:** [March 6, 2023, 4:51pm UTC](https://internals.rust-lang.org/t/micro-rfc-string-from-utf8-with-for-different-handling-of-invalid-utf8/18464/18 "2023-03-06T16:51:43Z")

</div>

Ah, I just misunderstood the below quote, thought you were suggesting somehow adding the function from `this` proposal to the `bstr` crate. But this makes more sense - I'll write up an ACP that goes this route.

> I would rather see a function like this added to std: [decode\_utf8 in bstr - Rust](https://docs.rs/bstr/latest/bstr/fn.decode_utf8.html)

---

<div class="post-metadata">

**Author:** ![zackw](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/zackw/32/2071_2.png) [@zackw](https://internals.rust-lang.org/u/zackw)\
**Post date:** [March 6, 2023, 5:16pm UTC](https://internals.rust-lang.org/t/micro-rfc-string-from-utf8-with-for-different-handling-of-invalid-utf8/18464/19 "2023-03-06T17:16:00Z")

</div>

Leaving aside the question of whether this belongs in the stdlib (now or eventually), I have some feedback on the API:

- Instead of having an enum, and a callback function signature, both of which implement a trait, wouldn't it be simpler to have a callback function signature, plus a module (if in the stdlib, something like `std::str::encoding_errors` perhaps) containing a bunch of free functions with that signature?

- The callback function signature you suggested doesn't seem right to me. Shouldn't it be something more like this?

- Naming bikeshed: I think `..._lossy_with` is not a very helpful name suffix and suggest `..._with_recovery` instead.

---

<div class="post-metadata">

**Author:** ![tgross35](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/tgross35/32/9550_2.png) [@tgross35](https://internals.rust-lang.org/u/tgross35)\
**Post date:** [March 6, 2023, 5:55pm UTC](https://internals.rust-lang.org/t/micro-rfc-string-from-utf8-with-for-different-handling-of-invalid-utf8/18464/20 "2023-03-06T17:55:06Z")

</div>

> - Instead of having an enum, and a callback function signature, both of which implement a trait, wouldn't it be simpler to have a callback function signature, plus a module (if in the stdlib, something like `std::str::encoding_errors` perhaps) containing a bunch of free functions with that signature?

That's also a possibility, and would be more or less what the enum would have to desugar to. I just couldn't think of anywhere in `std` that uses this kind of module, so it seemed like an enum was a more concise way of namespacing.

> The callback function signature you suggested doesn't seem right to me. Shouldn't it be something more like this?
> 
> ```rust
> type ToUnicodeErrorHandler =
> Fn(&'a [u8], FnMut(char)) -> NonZeroUsize;
> 
> ```

This seems a bit like the inverse of `bstr::decode_utf8`, and be nice to avoid copying. Perhaps something `impl Write` would be more idiomatic? This wouldn't allow for non-allocating implementations though...

As a nit, I'd think it would be better if the `&[u8]` points just to the invalid chunk rather than the entire slice, so the user doesn't have to use other utf8 logic to determine where the invalid slice ends.

Hm... that does remind me that there could likely be a `fn(&mut [u8]) -> usize` version to work with `&mut [u8]`, which would allow for `Ignore` or same-size replacements without allocating.

> - Naming bikeshed: I think `..._lossy_with` is not a very helpful name suffix and suggest `..._with_recovery` instead.

Agreed, especially since they're not always actually lossy. Maybe `_with` is enough since it's likely easy enough to infer. I'll update the main post

[Next page](https://internals.rust-lang.org/t/micro-rfc-string-from-utf8-with-for-different-handling-of-invalid-utf8/18464.md?page=2)