# PathBuf to CString **URL:** \ **Category:** libs\ **Created:** [June 15, 2020, 11:30am UTC](https://internals.rust-lang.org/t/pathbuf-to-cstring/12560 "2020-06-15T11:30:36Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![kornel](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/kornel/32/2711_2.png) [@kornel](https://internals.rust-lang.org/u/kornel)\ **Post date:** [June 15, 2020, 11:30am UTC](https://internals.rust-lang.org/t/pathbuf-to-cstring/12560/1 "2020-06-15T11:30:36Z")
Conversion from a `Path` to a C-compatible string is not easy in libstd. Because the conversion is not well-defined on Windows, access to bytes is hidden behind `cfg(unix)`. However, that means that in user code the correct conversion code for Unix also has to be behind `cfg(unix)`, and requires writing a fallback implementation. That is a lot to ask for such a minor detail, so people end up writing `CString::new(path.to_str().unwrap())` which is the worst case of all. I suggest adding `TryFrom` which preserves bytes on Unix, and falls back to UTF-8 on Windows. On Windows there's no right way to convert Unicode path to bytes. ANSI code pages could be useful in some cases, but they're a lossy legacy encoding and make the conversion environment-specific. OTOH sticking to UTF-8 wouldn't be worse than the current practice of `.to_str().unwrap()`. ---
**Author:** ![Riateche](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/riateche/32/1717_2.png) [@Riateche](https://internals.rust-lang.org/u/Riateche)\ **Post date:** [June 15, 2020, 11:54am UTC](https://internals.rust-lang.org/t/pathbuf-to-cstring/12560/2 "2020-06-15T11:54:59Z")
The lack of conversion from `Path` to `CString` is really inconvenient, so I would also like to see a solution in std or in a well-established crate. However, I don't think defaulting to UTF-8 on Windows is a good solution. This is a problematic conversion, so it's better to force users to make an explicit decision about the approach. Ideally, when passing a path to a library with C interface, you need to examine the library's documentation or source code to figure out which encoding it expects to receive. For example, some libraries explicitly declare that input must be UTF-8, and some libraries need the path to be converted to the current ANSI code page or OEM code page because they just call `fopen` with it. I think it makes sense to have a separate functions for these cases (and maybe there are other common cases). It seems that Windows API provides functions for this kind of conversion, so hopefully it can be done without dealing with all possible encodings directly. ---
**Author:** ![kornel](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/kornel/32/2711_2.png) [@kornel](https://internals.rust-lang.org/u/kornel)\ **Post date:** [June 15, 2020, 12:38pm UTC](https://internals.rust-lang.org/t/pathbuf-to-cstring/12560/3 "2020-06-15T12:38:19Z")
If there was no solution to this in libstd, then there would be some motivation to use an external crate. But `.to_str().unwrap()` is an easy-but-incorrect way out, which makes it harder to convince people to use a crate for this. It would be nice if Clippy could detect `CString::new(path.to_str().unwrap())` antipattern, but AFAIK Rust has a policy of not picking favorite crates, so Clippy couldn't offer a fix for it if it's not in the standard library. If `TryInto` shouldn't be that opinionated about encoding, how about adding a method on `Path` that's explicit about encodings? e.g. `path.to_bytes_which_may_or_may_not_be_utf8()` ---
**Author:** ![gus](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/gus/32/2924_2.png) [@gus](https://internals.rust-lang.org/u/gus)\ **Post date:** [June 17, 2020, 3:51am UTC](https://internals.rust-lang.org/t/pathbuf-to-cstring/12560/4 "2020-06-17T03:51:34Z")
I had exactly the OP's question when writing some basic bindings for a C library. I don't know anything about Windows (and don't care to invest more than minimal effort in it tbh). What _should_ I do for the `#[cfg(windows)]` case in a Rust function that takes `Path` and needs to call a C function that takes a `char*` filename? Let's say for simplicity that the C function (eventually) just calls `fopen()` with that `char*` value. ---
**Author:** ![uranusjr](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/uranusjr/32/5578_2.png) [@uranusjr](https://internals.rust-lang.org/u/uranusjr)\ **Post date:** [June 17, 2020, 7:26am UTC](https://internals.rust-lang.org/t/pathbuf-to-cstring/12560/5 "2020-06-17T07:26:51Z")
One possibility for Windows is to convert the path to its [short path](https://docs.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-getshortpathnamew) form (aka the [8.3 filename](https://en.wikipedia.org/wiki/8.3_filename)) if possible. I believe this is always safe to be encoded with the current code page, and therefore has a guarenteed `CString` representation that should work for all reasonably implemented C code (read: I haven’t personally encountered anything that doesn’t). Note that the short path is not available (it can be disabled by the user), so a fallback is still needed. In which case there is really no right answer; a C API that takes path as `char *` on Windows is broken by design, and it’s impossible to tell what encoding is correct without reading the actual implementation. UTF-8 is as good a guess as any. ---
**Author:** ![robinm](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/robinm/32/6255_2.png) [@robinm](https://internals.rust-lang.org/u/robinm)\ **Post date:** [June 17, 2020, 7:39am UTC](https://internals.rust-lang.org/t/pathbuf-to-cstring/12560/6 "2020-06-17T07:39:42Z")
> [@kornel](#): > > It would be nice if Clippy could detect `CString::new(path.to_str().unwrap())` antipattern, but AFAIK Rust has a policy of not picking favorite crates, so Clippy couldn't offer a fix for it if it's not in the standard library. Could this be solved by creating a new clippy-community linter that would suggest non-std crates, while not being "official"? ---
**Author:** ![jstarks](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/jstarks/32/6296_2.png) [@jstarks](https://internals.rust-lang.org/u/jstarks)\ **Post date:** [June 17, 2020, 6:44pm UTC](https://internals.rust-lang.org/t/pathbuf-to-cstring/12560/7 "2020-06-17T18:44:18Z")
Even on Windows, it would sometimes be nice to get the underlying (WTF-8) bytes, if only to interop with external code that uses WTF-8 instead of UCS-2 encoding. At the moment the only way to do that is to use `encode_wide` and then convert that back to WTF-8, which is quite wasteful. ---
**Author:** ![hyeonu](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/hyeonu/32/2162_2.png) [@hyeonu](https://internals.rust-lang.org/u/hyeonu)\ **Post date:** [June 20, 2020, 3:56am UTC](https://internals.rust-lang.org/t/pathbuf-to-cstring/12560/8 "2020-06-20T03:56:06Z")
Currently the underlying WTF-8 format is just an implementation detail and not exposed to any public interface. WTF-8 is not a standardised encoding but a known hack for handling encodings. Should we, the Rust language, define and manage a sort-of text encoding named WTF-8? ---
**Author:** ![mbrubeck](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/mbrubeck/32/174_2.png) [@mbrubeck](https://internals.rust-lang.org/u/mbrubeck)\ **Post date:** [June 20, 2020, 4:11am UTC](https://internals.rust-lang.org/t/pathbuf-to-cstring/12560/9 "2020-06-20T04:11:42Z")
> [@hyeonu](#): > > Should we, the Rust language, define and manage a sort-of text encoding named WTF-8? WTF-8 does have a [specification](https://simonsapin.github.io/wtf-8/), at least. Caveat: The specification requires that: > Any WTF-8 data _must_ be converted to a Unicode encoding at the system’s boundary before being emitted. UTF-8 is recommended. WTF-8 _must not_ be used to represent text in a file format or for transmission over the Internet. However, it's not clear how this is possible in general, because WTF-8 encodes a superset of Unicode. > [@jstarks](#): > > Even on Windows, it would sometimes be nice to get the underlying (WTF-8) bytes, if only to interop with external code that uses WTF-8 instead of UCS-2 encoding. There are some cases that would benefit from this even if they don't send the WTF-8 bytes to external code. For example, if you [use serde for IPC](https://crates.io/crates/ipc-channel), then sending a `Path` or `OsStr` between processes is currently much less efficient on Windows than non-Windows platforms. ---
**Author:** ![kornel](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/kornel/32/2711_2.png) [@kornel](https://internals.rust-lang.org/u/kornel)\ **Post date:** [June 23, 2020, 1:28pm UTC](https://internals.rust-lang.org/t/pathbuf-to-cstring/12560/10 "2020-06-23T13:28:44Z")
WTF-8 is supposed to be an internal representation, not an encoding for interchange. I don't think it's commonly used, so libraries expecting `char *` are very unlikely to make a proper use of it. Windows itself certainly can't. That makes exposing it more of a new feature for Rust rather than compatibility for C, so I'd say it's out of scope for `CString`. ---
**Author:** ![droundy](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/droundy/32/2054_2.png) [@droundy](https://internals.rust-lang.org/u/droundy)\ **Post date:** [July 1, 2020, 3:54pm UTC](https://internals.rust-lang.org/t/pathbuf-to-cstring/12560/11 "2020-07-01T15:54:36Z")
What about simply failing on Windows? If there is no correct behavior, that would ensure that code using `TryFrom` works where the result is well defined. ---
**Author:** ![kornel](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/kornel/32/2711_2.png) [@kornel](https://internals.rust-lang.org/u/kornel)\ **Post date:** [July 1, 2020, 8:07pm UTC](https://internals.rust-lang.org/t/pathbuf-to-cstring/12560/12 "2020-07-01T20:07:53Z")
Yeah, I suppose failing on any non-ASCII chars would be the most correct (I hope we can ignore EBCDIC :)) ---
**Author:** ![kornel](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/kornel/32/2711_2.png) [@kornel](https://internals.rust-lang.org/u/kornel)\ **Post date:** [July 9, 2020, 1:14pm UTC](https://internals.rust-lang.org/t/pathbuf-to-cstring/12560/13 "2020-07-09T13:14:47Z")
I shall add that this problem exists in `rustc` itself. LLVM takes `char*` for paths, and `rustc` uses things like: ```rust CString::new(format!("{}", path.display()) CString::new(path_buf.to_string_lossy().as_bytes()) ``` which is incorrect on all platforms. It's unfixable on Windows, but it's unnecessarily broken on Unix. ---
**Author:** ![jethrogb](https://avatars.discourse-cdn.com/v4/letter/j/43a26b/32.png) [@jethrogb](https://internals.rust-lang.org/u/jethrogb)\ **Post date:** [July 9, 2020, 1:15pm UTC](https://internals.rust-lang.org/t/pathbuf-to-cstring/12560/14 "2020-07-09T13:15:20Z")
The correct behavior on Windows is probably to convert the (wide) OsString to a multibyte string using [`WideCharToMultiByte`](https://docs.microsoft.com/en-us/windows/win32/api/stringapiset/nf-stringapiset-widechartomultibyte) with the active code page and check that `lpUsedDefaultChar` was not set. But this really depends on the C library. Using `char*` on Windows to refer to paths is not really portable. See notes regarding this in [Microsoft's `fopen` documentation](https://docs.microsoft.com/en-us/cpp/c-runtime-library/reference/fopen-wfopen?view=vs-2019#remarks). ---
**Author:** ![The\_Decryptor](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/the_decryptor/32/7027_2.png) [@The\_Decryptor](https://internals.rust-lang.org/u/The_Decryptor)\ **Post date:** [July 19, 2020, 2:46am UTC](https://internals.rust-lang.org/t/pathbuf-to-cstring/12560/15 "2020-07-19T02:46:29Z")
Microsoft are working on the portability issues, recent versions of Windows 10 support a flag that makes the system ANSI APIs consume and emit UTF-8. [https://docs.microsoft.com/en-us/windows/uwp/design/globalizing/use-utf8-code-page](https://docs.microsoft.com/en-us/windows/uwp/design/globalizing/use-utf8-code-page) The page says this will be the default going forward for new APIs as well, though of course support for that in third party libraries will be spotty, and it doesn't help old OS releases so care will still have to be taken. ---
**Author:** ![josh](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/josh/32/5934_2.png) [@josh](https://internals.rust-lang.org/u/josh)\ **Post date:** [July 20, 2020, 6:21am UTC](https://internals.rust-lang.org/t/pathbuf-to-cstring/12560/16 "2020-07-20T06:21:19Z")
Has to start somewhere; perhaps one day, when the minimum supported version of Windows is at least that version of Windows 10, we can just support UTF-8 for system APIs, and stop dealing with wide characters entirely except for conversions for FFI. ---
**Author:** ![matklad](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/matklad/32/12266_2.png) [@matklad](https://internals.rust-lang.org/u/matklad)\ **Post date:** [October 4, 2020, 6:18pm UTC](https://internals.rust-lang.org/t/pathbuf-to-cstring/12560/17 "2020-10-04T18:18:02Z")
It's interesting that all those `CString::new` also have `.unwrap` in them, which I think is impossible (as path can't have internal null bytes). Should we add ```rust pub trait std::os::unixOsStrExt { fn to_cstring(&self) -> CString; } ``` ? This obviously doesn't solve cross-platformness issue, but I am somewhat skeptical about that: it seems that windows "OS Strings" are composed of wide chars, and C strings of chars, and that seems fundamentally incompatible. ---
**Author:** ![bjorn3](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/bjorn3/32/2736_2.png) [@bjorn3](https://internals.rust-lang.org/u/bjorn3)\ **Post date:** [October 4, 2020, 7:01pm UTC](https://internals.rust-lang.org/t/pathbuf-to-cstring/12560/18 "2020-10-04T19:01:46Z")
On Windows `OSString` internally stores the data as WTF-8. This is UTF-8 compatible except for also representing unmatched surrogates. `CString` allows any byte sequence that doesn't contain a null, so storing WTF-8 in a `CString` is technically valid. ---
**Author:** ![matklad](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/matklad/32/12266_2.png) [@matklad](https://internals.rust-lang.org/u/matklad)\ **Post date:** [October 4, 2020, 7:22pm UTC](https://internals.rust-lang.org/t/pathbuf-to-cstring/12560/19 "2020-10-04T19:22:55Z")
I mean, it doesn't matter what internal format we use. Windows API uses `wchar_t *`, C API uses `char *`, they are just not interconvertable. ---
**Author:** ![chrisd](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/chrisd/32/7232_2.png) [@chrisd](https://internals.rust-lang.org/u/chrisd)\ **Post date:** [October 4, 2020, 7:35pm UTC](https://internals.rust-lang.org/t/pathbuf-to-cstring/12560/20 "2020-10-04T19:35:26Z")
> [@matklad](#): > > It's interesting that all those `CString::new` also have `.unwrap` in them, which I think is impossible (as path can't have internal null bytes). Wait but Rust `Path`s can. They are essentially just a bunch of bytes, no? Interpreting them as anything else depends on the particular function being called and the platform it is being called on. [Next page](https://internals.rust-lang.org/t/pathbuf-to-cstring/12560.md?page=2)