# PreRFC: trait converting functions for OsString **URL:** \ **Category:** libs\ **Created:** [January 14, 2020, 8:35pm UTC](https://internals.rust-lang.org/t/prerfc-trait-converting-functions-for-osstring/11634 "2020-01-14T20:35:29Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![Nokel81](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/nokel81/32/3966_2.png) [@Nokel81](https://internals.rust-lang.org/u/Nokel81)\ **Post date:** [January 14, 2020, 8:35pm UTC](https://internals.rust-lang.org/t/prerfc-trait-converting-functions-for-osstring/11634/1 "2020-01-14T20:35:30Z")
Given that `env::var_os` produces a `std::ffi::OsString` (as it should) it would be nice to be able to write functions that rely on the implementation of `TryFrom`. Now, for most types this would be best done by hand, but I think a simpler stop gap could be to have `impl TryFrom for String` since if I remember correctly, `from` and `try_from` can be executed several times. ---
**Author:** ![CAD97](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/cad97/32/3460_2.png) [@CAD97](https://internals.rust-lang.org/u/CAD97)\ **Post date:** [January 14, 2020, 11:13pm UTC](https://internals.rust-lang.org/t/prerfc-trait-converting-functions-for-osstring/11634/2 "2020-01-14T23:13:05Z")
> [@Nokel81](#): > > since if I remember correctly, `from` and `try_from` can be executed several times. [`TryFrom::try_from`](https://doc.rust-lang.org/std/convert/trait.TryFrom.html) takes the argument by value, so that exact impl would be only usable once per `OsString`. That said, it also matches the signature of `OsString::into_string`, so is a sensible impl. ```rust impl TryFrom for String { type Error = OsString; fn try_from(value: OsString) -> Result { value.into_string() } } ``` ---
**Author:** ![Nokel81](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/nokel81/32/3966_2.png) [@Nokel81](https://internals.rust-lang.org/u/Nokel81)\ **Post date:** [January 15, 2020, 1:48pm UTC](https://internals.rust-lang.org/t/prerfc-trait-converting-functions-for-osstring/11634/3 "2020-01-15T13:48:40Z")
Makes sense, this probably can be just a PR. However, it does seem weird that `OsString` cannot be converted to/from String directly (or by checking to see if it is already valid). Was that intentional? ---
**Author:** ![Tom-Phinney](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/tom-phinney/32/3299_2.png) [@Tom-Phinney](https://internals.rust-lang.org/u/Tom-Phinney)\ **Post date:** [January 15, 2020, 2:21pm UTC](https://internals.rust-lang.org/t/prerfc-trait-converting-functions-for-osstring/11634/4 "2020-01-15T14:21:37Z")
`string` is not platform-specific, whereas `OsString` inherently is. Perhaps that is the reason. ---
**Author:** ![Nokel81](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/nokel81/32/3966_2.png) [@Nokel81](https://internals.rust-lang.org/u/Nokel81)\ **Post date:** [January 15, 2020, 2:24pm UTC](https://internals.rust-lang.org/t/prerfc-trait-converting-functions-for-osstring/11634/5 "2020-01-15T14:24:27Z")
Yes, I guess there was a desire not to have to implement the conversion per platform (for some definition of platform). ---
**Author:** ![burntsushi](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/burntsushi/32/279_2.png) [@burntsushi](https://internals.rust-lang.org/u/burntsushi)\ **Post date:** [January 15, 2020, 2:42pm UTC](https://internals.rust-lang.org/t/prerfc-trait-converting-functions-for-osstring/11634/6 "2020-01-15T14:42:55Z")
That's what `into_string()` does. On Unix, an `OsString` can be an arbitrary sequence of bytes, which is not necessarily valid UTF-8. On Windows, an `OsString` is a WTF-8 encoding of an arbitrary sequence of 16-bit integers, which also may not be valid UTF-8. ---
**Author:** ![Nokel81](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/nokel81/32/3966_2.png) [@Nokel81](https://internals.rust-lang.org/u/Nokel81)\ **Post date:** [January 15, 2020, 2:55pm UTC](https://internals.rust-lang.org/t/prerfc-trait-converting-functions-for-osstring/11634/7 "2020-01-15T14:55:51Z")
That isn't what I get from either the documentation or the method signature. I get that `into_string()` merely checks that it is a valid `std::string::String` and nothing more. It doesn't do any translation between WTF-8 and UTF-8 for instance. ---
**Author:** ![chrisd](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/chrisd/32/7232_2.png) [@chrisd](https://internals.rust-lang.org/u/chrisd)\ **Post date:** [January 15, 2020, 3:12pm UTC](https://internals.rust-lang.org/t/prerfc-trait-converting-functions-for-osstring/11634/8 "2020-01-15T15:12:49Z")
WTF-8 is a strict superset of UTF-8. With valid unicode, there's no conversion necessary. When call Windows APIs there is a conversion to u16 integers. Any string results from the API are then immediately converted to WTF-8. ---
**Author:** ![burntsushi](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/burntsushi/32/279_2.png) [@burntsushi](https://internals.rust-lang.org/u/burntsushi)\ **Post date:** [January 15, 2020, 3:13pm UTC](https://internals.rust-lang.org/t/prerfc-trait-converting-functions-for-osstring/11634/9 "2020-01-15T15:13:07Z")
I didn't say it did any translation? I'm explaining why "However, it does seem weird that `OsString` cannot be converted to/from String directly" isn't possible. Because `String` is a strictly more restrictive type than `OsString`. ---
**Author:** ![Nokel81](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/nokel81/32/3966_2.png) [@Nokel81](https://internals.rust-lang.org/u/Nokel81)\ **Post date:** [January 15, 2020, 3:34pm UTC](https://internals.rust-lang.org/t/prerfc-trait-converting-functions-for-osstring/11634/10 "2020-01-15T15:34:01Z")
That is something that I did not know. However, in the docs of `std::ffi::OsString` is says that on windows it is composed of 16 bit values that may be interpreted as UTF-16 where applicable. However, from [this stackoverflow](https://stackoverflow.com/questions/32499846/is-utf-16-compatible-with-utf-8) post, it seems to be the case that UTF-16 is not a superset of UTF-8. This is main reason for my confusion since it seems that if the UTF-16 data is not UTF-8 then there is no standard way to convert it to be a UTF-8 string. ---
**Author:** ![chrisd](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/chrisd/32/7232_2.png) [@chrisd](https://internals.rust-lang.org/u/chrisd)\ **Post date:** [January 15, 2020, 3:45pm UTC](https://internals.rust-lang.org/t/prerfc-trait-converting-functions-for-osstring/11634/11 "2020-01-15T15:45:08Z")
To be clear there are four different encodings here: - Windows Strings - UTF-16 - UTF-8 - WTF-8 UTF-16 and UTF-8 are different encodings for the same unicode standard. So you can always convert between UTF-16 and UTF-8 without losing anything. Windows strings are not unicode. They are an array of `u16` integers which may or may not be UTF-16. WTF-8 is not unicode. It's an array of `u8` integers that may or may not be valid UTF-8.. ---
**Author:** ![Nokel81](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/nokel81/32/3966_2.png) [@Nokel81](https://internals.rust-lang.org/u/Nokel81)\ **Post date:** [January 15, 2020, 6:08pm UTC](https://internals.rust-lang.org/t/prerfc-trait-converting-functions-for-osstring/11634/12 "2020-01-15T18:08:41Z")
Yes that is what my understanding is. My quandary is that there doesn't seem to be a way to convert a windows string (as an `OsString`) to a valid utf-8 rust string. Even if it is "assume utf-16, then convert". ---
**Author:** ![Tom-Phinney](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/tom-phinney/32/3299_2.png) [@Tom-Phinney](https://internals.rust-lang.org/u/Tom-Phinney)\ **Post date:** [January 15, 2020, 6:22pm UTC](https://internals.rust-lang.org/t/prerfc-trait-converting-functions-for-osstring/11634/13 "2020-01-15T18:22:24Z")
There is no way to _interpret_ a UTF-16 string as a UTF-8 string, because the bit patterns in memory are different. However, every UTF-16 string can be recoded as a UTF-8 string, and vice versa. ---
**Author:** ![burntsushi](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/burntsushi/32/279_2.png) [@burntsushi](https://internals.rust-lang.org/u/burntsushi)\ **Post date:** [January 15, 2020, 6:31pm UTC](https://internals.rust-lang.org/t/prerfc-trait-converting-functions-for-osstring/11634/14 "2020-01-15T18:31:53Z")
> [@Nokel81](#): > > there doesn't seem to be a way to convert a windows string (as an `OsString` ) to a valid utf-8 rust string That is exactly what [`OsString::into_string`](https://doc.rust-lang.org/std/ffi/struct.OsString.html#method.into_string) is. If the internal WTF-8 happens to be UTF-8, then you get a `String` back. If the internal WTF-8 is not valid UTF-8 (which means the original Windows string contained invalid UTF-16), then you get the `OsString` back. At that point, your only recourse is to [lossily decode the OS string](https://doc.rust-lang.org/std/ffi/struct.OsStr.html#method.to_string_lossy) or otherwise [convert the internal WTF-8 back to the corresponding 16-bit integers](https://doc.rust-lang.org/std/os/windows/ffi/trait.OsStrExt.html#tymethod.encode_wide) and do what you need there. I think there might be some confusion here, so if this doesn't clear things up, then I'd recommend posting a concrete example that shows the conversion you want. ---
**Author:** ![chrisd](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/chrisd/32/7232_2.png) [@chrisd](https://internals.rust-lang.org/u/chrisd)\ **Post date:** [January 15, 2020, 6:58pm UTC](https://internals.rust-lang.org/t/prerfc-trait-converting-functions-for-osstring/11634/15 "2020-01-15T18:58:05Z")
Here's some wrapper functions for each conversion between types. ```rust /// Assumes strings are valid unicode. May panic otherwise. use std::ffi::{OsString, OsStr}; // Only works on windows use std::os::windows::ffi::{OsStringExt, OsStrExt}; fn rust_to_windows(s: String) -> Vec { s.encode_utf16().collect() } fn rust_to_osstring(s: String) -> OsString { s.into() } fn osstring_to_rust(s: OsString) -> String { s.into_string().unwrap() } fn osstring_to_windows(s: OsString) -> Vec { s.encode_wide().collect() } fn windows_to_osstring(s: Vec) -> OsString { OsString::from_wide(&s) } fn windows_to_rust(s: Vec) -> String { String::from_utf16(&s).unwrap() } ``` Notice that the wrapper functions are simply calling built-in Rust functions. ---
**Author:** ![Nokel81](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/nokel81/32/3966_2.png) [@Nokel81](https://internals.rust-lang.org/u/Nokel81)\ **Post date:** [January 15, 2020, 7:15pm UTC](https://internals.rust-lang.org/t/prerfc-trait-converting-functions-for-osstring/11634/16 "2020-01-15T19:15:59Z")
Yes I know, that is why I was talking about "converting" not "interpreting" ---
**Author:** ![Nokel81](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/nokel81/32/3966_2.png) [@Nokel81](https://internals.rust-lang.org/u/Nokel81)\ **Post date:** [January 15, 2020, 7:21pm UTC](https://internals.rust-lang.org/t/prerfc-trait-converting-functions-for-osstring/11634/17 "2020-01-15T19:21:04Z")
I agree that there was confusion. And this reply has cleared most of it, so thank you. My last question is that [https://simonsapin.github.io/wtf-8/](https://simonsapin.github.io/wtf-8/) seems to imply that if a string is UTF-16 compliant it may not be WTF-8.... ---
**Author:** ![burntsushi](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/burntsushi/32/279_2.png) [@burntsushi](https://internals.rust-lang.org/u/burntsushi)\ **Post date:** [January 15, 2020, 8:27pm UTC](https://internals.rust-lang.org/t/prerfc-trait-converting-functions-for-osstring/11634/18 "2020-01-15T20:27:59Z")
I think you are mixing up what "encoding" means. UTF-8, WTF-8 and UTF-16 are all encodings. UTF-8 and UTF-16 are ways of representing Unicode code points in a raw byte representation. UTF-8 uses between 1 and 4 bytes for each codepoint. UTF-16 uses either 2 bytes or 4 bytes for each codepoint. UTF-8 cannot represent all file paths on UNIX because file paths are an arbitrary sequence of bytes (without interior NUL bytes). Similarly, UTF-16 cannot represent all file paths on Windows because file paths are an arbitrary sequence of 16-bit integers. WTF-8 was invented to bridge the gap such that file paths on Windows can be non-lossily roundtripped between Rust's `OsString` and `PathBuf` types. WTF-8 was chosen so that Rust `String` types could be used in a zero cost fashion to manipulate file paths. (e.g., `file_path.join("foo")`.) When a `PathBuf` is used on Windows to interact with the operating system, it is converted to an `OsString` and then transcoded from its internal WTF-8 encoding (a strict superset of UTF-8) to a sequence of 16-bit integers that Windows expects. When you have a `PathBuf` or an `OsString` in Rust _in memory_, its internal representation in memory uses WTF-8. When you actually go to use it for anything outside of Rust, it is first transcoded to 16-bit integers. Since WTF-8 is a strict superset of UTF-8 and can otherwise encode all possible sequences of 16-bit integers, it follows that 1) OsStrings can be manipulated in a zero cost fashion with Rust's guaranteed valid UTF-8 encoded `String` types and that 2) all possible file paths on Windows can be correctly represented and roundtripped by Rust. The cost is that, at least on Windows, all file paths must be transcoded to and from WTF-8 and its sequence of arbitrary 16-bit integers. If a file path on Windows contains valid UTF-16, then its corresponding WTF-8 representation is guaranteed to be valid UTF-8. The WTF-8 representation is only invalid UTF-8 in precisely the case where the file path on Windows is not valid UTF-16. In which case, you get a WTF-8 encoded string that cannot be converted to UTF-8 without error or replacement or omission. ---
**Author:** ![Nokel81](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/nokel81/32/3966_2.png) [@Nokel81](https://internals.rust-lang.org/u/Nokel81)\ **Post date:** [January 15, 2020, 8:46pm UTC](https://internals.rust-lang.org/t/prerfc-trait-converting-functions-for-osstring/11634/19 "2020-01-15T20:46:39Z")
Thanks for the clarification. I think the lack of knowledge of `std::os::windows::ffi` was also contributing to my misunderstanding ---
**Author:** ![system](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/system/32/14092_2.png) [@system](https://internals.rust-lang.org/u/system)\ **Post date:** [April 14, 2020, 8:46pm UTC](https://internals.rust-lang.org/t/prerfc-trait-converting-functions-for-osstring/11634/20 "2020-04-14T20:46:43Z")
This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.