[Pre-RFC] Additional path handling utilities

[Screwtapello]

Finally, I think relying on the current working directory for anything related to generate relative paths is not a good idea. (OTOH, I don’t totally understand the reasoning around needing to get the CWD, so I might be missing something.) The relative_path_from method in ruby’s Pathname class has worked well for my use-cases.

Computing the relative path from one filesystem location to another fundamentally requires two absolute paths - if you’re not talking about two specific locations, then you can’t possibly navigate reliably from one to the other. If you’re using a library like @soc talks about, where there’s a data-type with the “is an absolute path” invariant, this is easy—you just put the method on that data type. However, if you’ve only got one data-type for both absolute and relative paths (as is the case with PathBuf), one or both of the inputs might be relative, and you just have to deal with it, so what do you do?

One solution is to return an error if one or both paths are relative, which is simple, predictable behaviour but a bit annoying.

Another solution is to say “relative paths are relative to the current directory”, which has some precedent, and seems to be what’s being proposed for Path::relative_to().

I’m not familiar with Ruby’s relative_path_from(), but the docs mention that either both inputs must be absolute, or both must be relative, which I suspect means it gives the same result as concatenating paths with the current working directory (or any fixed path, like /).

[ExpHP]

Update on this: Sorry for getting your hopes up. I was wrong. Turns out that file was already there in my home directory.

I really coulda sworn, man…

[Screwtapello]

Bah humbug. The semantics of UNIX symlinks are horrible from a user perspective. When I cd into some symlink into my home directory and then cd .. , I intend to go home.

The semantics of symlinks may be subtle and quick to anger, but they are what they are. I don’t mind tools designed for human interaction like shells and GUI file-managers doing a bit of magic behind the scenes (bash does this for the cd command by default, you can turn it off with set -P), but I’m uneasy about that kind of “do what I mean” functionality lower in the stack.

Basically, my nightmare scenario is being woken up at 3AM because of a production outage, and the system logs say it read a bogus value from a file at a particular path, and that path doesn’t actually exist on disk because it’s the “cleaned” version of the path that was really used, and there’s no good way to find out what actually happened.

When I’m trying to understand and diagnose a failure, I don’t mind “confusing” or “unfriendly” or “legacy” behaviour because I can trace through that if I have to, as long as I have all the details available. I dislike approximations and heuristics, even ones that attempt to be helpful and guiding, because there’s a chance they’ll throw away the clue I needed to understand what was going on.

[ExpHP]

There’s a time and place for everything. I would consider that for these specific situations:

• Paths coming from CLI arguments
• When the program itself creates a tree of directories and symlinks in the filesystem, and constructs paths into that tree

cleaning (i.e. logical path manipulation) is the thing that will give you the information you care about. I mean, what if the issue was that the program created a symlink pointing to the wrong directory? It’d take you forever to find that out if it only told you canonicalized paths.

Meanwhile, for these specific situations:

• a path contained somewhere in some file (a config file, a source file, whatever)
• …actually, yeah, I guess that’s it, but it’s a pretty big one.

canonicalized paths clearly win out.

---

My point is: A tree can fall in more than one direction.

…

(hmm… bad analogy, sometimes trees are supposed to fall. Maybe “a bridge can collapse from either side”?)

[ExpHP]

Not to mention: The standard library’s fascination with canonical paths (which at some point plagued the path_abs crate to an even greater extent) is terrifying if you ever want to delete something. (which could be a link)

[soc]

It’s interesting that in Java, despite having an explicit LinkOption enum, Files.delete(Path) doesn’t accept such an argument, and always opts for deleting the symbolic link.

I think Rust’s documentation on remove_file is insufficient regarding its semantics: https://doc.rust-lang.org/std/fs/fn.remove_file.html

Maybe its behavior could be derived by thinking about the semantics of unlink, but I think the behavior on links should be documented.

[kornel]

append/push/join/concat creates lots of names for almost the same thing. That’s kinda confusing.

Also does any one of them provide safety from path traversal vulnerabilities? I need root_dir.safely_join(subpath) to never be able exit root_dir (joining ../../etc should be an error or join ./etc instead).

[soc]

Also does any one of them provide safety from path traversal vulnerabilities?

That’s a very good point! The path API felt like it had a very scripting-oriented design until now, were people are intended to do stuff, and then hopefully check later on whether the stuff they did returned the intended result.

I think it would be a useful addition – though I share the concern that good names are hard to find.

Maybe it makes sense to have a safe-by-default approach (ok, that’s a hard sell in the presence of join and push :-D) and prohibit path traversal in append and concat, and have unsafe_append and unsafe_concat in addition?

[uberjay]

Taking a quick survey, I like the names used elsewhere a bit better – join has the problem that it feels very… uh… mechanical? Maybe because of the conflict with slice join? I have a suspicion the intuitive understanding of path join is that it’s like slice join, except the separator is chosen for you. Apparently C# calls their method Combine. (and it has the same behavior as PathBuf.join). But, a function named Combine hints that there may be more going on than just mechanical string concatenation.

I actually like the names of the javascript path maniuplators:

join - does not interpret input paths as possibly absolute.

resolve - does allow for input absolute path.

Again, “resolve” tells me there’s more going on with the operation. It’s even described in a totally different way – it’s not that an input path can “reset” or “destroy” the existing path:

The given sequence of paths is processed from right to left, with each subsequent path prepended until an absolute path is constructed. For instance, given the sequence of path segments: /foo , /bar , baz , calling path.resolve('/foo', '/bar', 'baz') would return /bar/baz .

[uberjay]

Update on this: Sorry for getting your hopes up. I was wrong. Turns out that file was already there in my home directory.

I really coulda sworn, man…

d’oh!!! That is a bummer! It sounded plausible, but also, perhaps a bit too magical for my tastes.

[scottmcm]

Path::relative_to and Path::absolute

Path would also gain two more functions for computing relative paths to a base one, as well as the absolute base path for any given relative one. These functions require the current working directory (CWD) to be available. Although in cases where Path::starts_with is true, relative_to would not require it.

I’m a big fan of relative_to; I remember discussing it for Boost in C++ (wow, 10 years ago already?).

But it needs more than just the CWD, because it needs to be symlink-aware to work in general.

One solution is to return an error if one or both paths are relative, which is simple, predictable behaviour but a bit annoying.

This seems eminently reasonable to me. It’s a place where it’s easy to look-before-you-leap if you need to, and you can always explicitly use a cwd-aware function to make a path absolute before hand if you need to.

The difference is that it doesn’t actually touch the filesystem.

I wonder if there’s a place for some sort of domain separation parameter in paths, or association with a VFS, or something? Because in a sense the difference here is between a LocalPath and an AbstractPath, and maybe there’s space for things like an InMyZipFilePath and such. Also, perhaps an AbstractPath isn’t based on OsStr, which would mean it can’t be a newtype around or method on Path…

Is there something commonly used for the Path component of URLs in rust?

[Screwtapello]

There’s a time and place for everything. I would consider that for these specific situations:

• Paths coming from CLI arguments
• When the program itself creates a tree of directories and symlinks in the filesystem, and constructs paths into that tree

cleaning (i.e. logical path manipulation) is the thing that will give you the information you care about.

For a CLI program that uses its arguments directly, like grep or find, I wouldn’t bother with any cleaning or canonicalisation at all. If the user requests an operation on “foo”, it’s easiest for them to understand results and responses if they’re phrased in identical terms.

As for the directory-tree-as-data-structure case, I came up with my “monotonic” algorithm while I was working on a crate for local http caching, so it’s definitely the way I want such a program to work.

I mean, what if the issue was that the program created a symlink pointing to the wrong directory? It’d take you forever to find that out if it only told you canonicalized paths.

That example cuts both ways, though: imagine a symlink pointing to the wrong place, and you get an error message saying “could not read foo/bar/baz: not found” when foo/bar/baz definitely exists and its last-modified time is much older than the error message.

Not to mention: The standard library’s fascination with canonical paths (which at some point plagued the path_abs crate to an even greater extent) is terrifying if you ever want to delete something. (which could be a link)

Yeah, a fully-canonicalized path is useful when you care about file contents but less so when you care about the directory structure itself. That said, the “monotonic” algorithm never tries to resolve a symlink at the end of a path (because the end of a path can never be followed by .. or it wouldn’t be the end of the path) so I’m not worried about that.

Maybe it makes sense to have a safe-by-default approach (ok, that’s a hard sell in the presence of join and push :-D) and prohibit path traversal in append and concat , and have unsafe_append and unsafe_concat in addition?

“unsafe” usually means exclusively memory-unsafety, so maybe not those particular names. This is more the “validated” versus “unvalidated” kind of safety, like SQL injection and cross-site-scripting, and (like those problems) the real solution is separate data-types that can’t be easily mixed. I think this thread is about extending the (single) PathBuf type rather than designing a new, safer path manipulation API, so I don’t think “safe by default” is a practical goal here.

On the other hand, providing the tools to build a safe, higher-level API seems reasonable. How would you feel about a is_relative_descendant(&self) -> bool method that returns true for a path that does not start with a prefix or a root component, and does not escape its prefix with .. components? It would fit nicely with is_absolute() and is_relative(), since on Windows, paths like \foo and C:foo are relative, but not relative descendants.

I wonder if there’s a place for some sort of domain separation parameter in paths, or association with a VFS, or something?

You might look at Python’s pathlib API, which has PurePosixPath and PureWindowsPath types, which do not touch the filesystem and can therefore be used on any platform, and PosixPath and WindowsPath types which do touch the filesystem and therefore can only be used on their respective platforms.

[Soni]

I don’t like how Rust paths are just type-punned strings.

most of the overhead is in the filesystem. I feel like you made a mistake by allowing strings to be type-punned into paths.

also, not every system uses string paths. some use string arrays and place no restrictions on filenames (except the NUL byte). it’s impossible to port Rust to those systems, as far as I can tell.

[vitiral]

I think this should mostly be fixed now, take a look at the new version and @Screwtapello is working on improving the story even more. We now do not require paths to exist and should be able to delete links

[Screwtapello]

If it’s not too late, I’d like to propose another utility method: head()

let mut p = Path::new(r"C:\Windows\System32").head();
assert_eq!(p, Path::new(r"C:\");

let mut p = Path::new(r"\Users").head();
assert_eq!(p, Path::new(r"\");

let mut p = Path::new(r"C:some\other\path").head();
assert_eq!(p, Path::new(r"C:");

let mut p = Path::new(r"pure\relative\path").head();
assert_eq!(p, Path::new(r"");

This method returns the “head” of a path; that is, the Prefix and RootDir components (if any). This is useful for implementing “adjoinment” logic (if you want to implement your own alternative to canonicalize()) and could also be useful for Windows programs want to know whether two paths are on the same drive.

For an absolute path, this is easily accomplished with p.push('/'), but that doesn’t work for a relative path if you want to keep it relative. You can repeatedly call .parent() until it returns None, but that repeats the hard work of segmenting the path into components for each call. You can walk over the components() iterator, stopping when you reach the first non-Prefix, non-RootDir component and collecting into a new PathBuf, but that requires a new allocation.

The .head() method suggests a companion .tail() method that skips Prefix and RootDir components and returns the part of the path composed of Normal, CurrentDir and ParentDir components, but I haven’t thought of a practical use-case for that yet.

[scottmcm]

If it’s not too late, I’d like to propose another utility method […]

It might be worth taking a look at what Boost.Filesystem does here:

https://www.boost.org/doc/libs/1_68_0/libs/filesystem/doc/reference.html#path-decomposition-table

[epage]

This method returns the “head” of a path; that is, the Prefix and RootDir components (if any). This is useful for implementing “adjoinment” logic (if you want to implement your own alternative to canonicalize() ) and could also be useful for Windows programs want to know whether two paths are on the same drive.

In Python’s pathlib, this is called .anchor.

[epage]

“unsafe” usually means exclusively memory-unsafety, so maybe not those particular names.

There is a suggestion in the API Guidelines to offer unchecked versions of functions that are unsafe. There is some question about invariant-safety vs memory-safety so I’ve opened rust-lang-nursery/api-guidelines#179

[killercup]

I don’t like how Rust paths are just type-punned strings.

That’s not the case: PathBuf is a newtype around OsString, which has different guarantees than String.

[Soni]

Due to the &str (or &OsStr at least) to Path and vice-versa AsRef conversions, it’s impossible to have Path be a NULL-terminated array of OsStr in platforms that use such representation.

This also applies to PathBuf.

AsRef = type-punned.