Revisiting modules, take 3

[petrochenkov]

Module names are used all the time outside of use and pub(restricted):

I mean, module names as full paths, not path prefixes.
let x = mod_name;, match x { mod_name => ... }, let _: mod_name; are always errors.
So, let x = [ident];, macth x { [ident] => ... }, let _: [ident]; can be unconditionally parsed as arrays/slices and not paths to a crate named ident.

[est31]

Can we please avoid deprecating mod? It doesn’t just force you to open an often very inconveniently long tree view of the file system, it also comes with disadvantages like not being able to make all contents of a module private except you do a reexport.

[rpjohnst]

Deprecating mod doesn’t have to mean either of those things. Python doesn’t have it, and people can find modules just fine based on their path. And, implicit modules can be private by default- that’s an independent change.

Personally I would prefer modules be private by default, included in the build based on use, and made visible outside via pub use.

[est31]

Python doesn’t have it, and people can find modules just fine based on their path.

I think navigation inside large codebases is not one of the the virtues of Python, while Rust specifically targets large codebases with its features like static typing and the ownership system, telling you many things about a function’s behaviour statically. Getting closer to python should be no goal.

And, implicit modules can be private by default- that’s an independent change.

Right now you can see when looking at a mod declaration whether a module is private or not (IMO reexports don’t change privacy of the reexported thing, as much as wrappers around functions don’t change it). If you go with a model where the privacy of a module is the privacy of its most public member, you won’t have this feature any more. You simply won’t know, is this module part of the public API? Is it some util thing?

If you go with a model like you suggest where you force modules to be private, and only allow public modules through doing pub use, you reintroduce pub mod with a new name.

[rpjohnst]

I think navigation inside large codebases is not one of the the virtues of Python

That is hardly because of its module system, which is far less confusing than Rust’s. A Python module’s position in the namespace is the same as its position in the filesystem, so when you see a path you can go straight to its definition.

If you go with a model like you suggest where you force modules to be private, and only allow public modules through doing pub use, you reintroduce pub mod with a new name.

Yes, that’s the point. One of the confusing parts of the module system is the redundancy between all the various module-related declaration keywords, and how to gain access to a new file in the first place. If we get rid of private mod foo; declarations and replace them with use foo;, that problem goes away.

[est31]

I wonder if there’s a way to (1) keep paths for use starting at crate root, (2) keep use as items, but (3) avoid the surprise when certain things work at crate root but not deeper within the crate.

I think most of the surprise comes from the external crate problem, so most of it will be fixed if you don’t mount external crates at the root, but manage them through [cratename] instead.

Especially, if you disallow stuff like fn mention_write<T>() where T: std::io::Write {} and require it to be written like in submodules to the root, aka with [std]::io::Write, I think most of the confusion will be gone.

AFAIK, currently, the only thing that is technically different in the crate root from other places of the crate is that std or core get auto-mounted at the root. In theory, you can declare extern crate in submodules as well so its the same except for std. However, popular practice is to put the extern crate declarations most times into the root, making the root practically different in this regard as well.

So the only difference you could think that remains where the crate root is different from one of its submodules is that it can use its modules with seemingly absolute paths (no need for :: prefixing) but that property is IMO already the case for any module and its submodules, and there is also no practical difference arising from this; if you want to split up the crate root, maybe you want to split up the first level of modules into a second level as well at a later point.

[arielb1]

As was discussed above, there’s also the problem that new users might form a model where paths in use behave like paths in expressions, and get confused "why can’t use find my local use". I think we can fix this by making paths in use work like paths in expressions (requiring use ::local::thing;).

[rpjohnst]

This has also been mentioned before, but that may not be the whole/actual problem, for a couple of reasons. Plenty of other languages use absolute paths in imports but relative paths elsewhere, without causing confusion. Instead the problem seems to be that, in Rust, this difference only exists in submodules. Further, a large portion of uses are absolute, so requiring a :: prefix would add a lot of tedium and extra syntax for (given the above) not much gain.

Taking crates out of the root namespace does only partially resolve the difference between the use in the root module and use in submodules. Using submodules is still different between the two. However: this difference is also present in these same other languages. Perhaps we simply need to teach use similarly- by analogy to include paths, $PATH, etc.

[withoutboats]

Python’s import statement is an interesting example that I think is pretty good, but hard to get to for us.

Imports are from an absolute root, which contains this package under its name:

import serde.Serialize
import thispackage.Item

extern and crate dependencies are distinguished then by you knowing the name of the crate you’re looking at.

Relative imports also work, using a leading period. However, in most modules these would be the equivalent of super::, whereas for __init__.py (the equivalent of mod.rs) they are equivalent to self::. This gives . the very understandable semantics of “from this directory.”:

import .sibling.Item # imports from $PWD/sibling.py

Problems with migrating to this model for us:

• Our crates can depend on a crate of the same name.
• Unclear how to move from self and super to a “this directory” signifier.

[withoutboats]

I didn’t respond to your comments earlier (my apologies), but I hadn’t been drawing a distinction between crate root and extern in my earlier analysis, and your numbers confirmed what I had believed, which was that self:: prefixed imports are a very small minority (on the order of 5% or less) of use statements, especially non-pub use statements. Between this and the fact that so many mainstream languages use absolute paths in imports (& I don’t actually know one that doesn’t), it seems clear that making self:: the default would be the wrong default.

[arielb1]

it seems clear that making self:: the default would be the wrong default.

That’s why I think that a combination of self::LOCAL/super::WHATEVER/::ABSOLUTE/[crate]::EXTERN is the best solution (with unprefixed paths in uses being deprecated).

[troplin]

self:: prefixed imports are a very small minority (on the order of 5% or less) of use statements, especially non-pub use statements

Do those numbers also include use statements that are absolute but could actually be relative?

I’ve often written use statements as they were relative, then found out that it does not work and used absolute paths instead because I didn’t remember that self:: existed. So my code actually contains more absolute imports than necessary and only very few uses of self::.

[RalfJung]

Python’s import statement is an interesting example that I think is pretty good, but hard to get to for us.

However, Python also doesn’t allow you to use things from another “crate” at all without importing, right? Paths inside the module are relative, and there is no “be absolute” modifier. That would be problematic in Rust at least for error messages, which may have to mention types that have not been imported but only come up due to inference.

Relative imports also work, using a leading period. However, in most modules these would be the equivalent of super::, whereas for __init__.py (the equivalent of mod.rs) they are equivalent to self::. This gives . the very understandable semantics of “from this directory.”:

Ah, I never thought about this. I didn’t even notice that relative imports work “differently” in __init__ vs. other files. Neat!

---

I wonder if there’s a way to (1) keep paths for use starting at crate root, (2) keep use as items, but (3) avoid the surprise when certain things work at crate root but not deeper within the crate.

So do you mean to rule out code like this, or to somehow make it less surprising that this stops working when moved into a submodule?

mod foo {
  fn bar() {}
}
use foo::bar;

fn main() {
  bar();
}

At this point we are talking about use mystuff vs. use ::mystuff, so the question is whether it is worth to have two additional characters at almost all crate-local uses to remind us that these are absolute. Or, phrased the other way around, is some weaker form of path confusion a price worth paying for saving two characters per crate-local use?

---

[std] is already a path to crate root, no need in the weird trailing ::.

use [std]; // Equivalent to `extern crate std;`

This actually would be a special case – the [std]:: prefix roots the path, but so far I have not seen as [std] as being an item on its own.

[aturon]

Just a small note to keep everybody in the loop:

• I’ve been talking some out of band with @withoutboats to sort through some of the issues I raised earlier in the thread, and to try to get on the same page with respect to a few knobs.
• Unsurprisingly, these conversations have led to yet further variants!
• At this point, where I think we’ve converged (relatively) on the broad outlines, I think the right next step is to write up the latest variant as a full-fledged RFC and move discussion there, where we can all work through the fine details together. This RFC should take into account the points that have been raised throughout the discussion.

@withoutboats is traveling at the moment, but is working on the writeup and will hopefully get it posted toward the end of this week.

Please note, this is not intended to cut off discussion, but rather to signal that this conversation has produced what seems like a viable candidate, and so we want to start the more formal RFC process.

[dan_t]

There have been quite a bit of discussions regarding modules here, so I’m sorry if couldn’t follow all ones and someone already had the same idea.

Personally I’m more in favour of only having absolute paths for use. I also hadn’t that much issues identifying external crates by just the first part of use (e.g. use clap::...).

I can see how this gets harder for foreign projects to distinguish between modules from external crates and internal modules.

What if the first part of use always had to be a crate name? So even in the case of importing internal modules you would’ve to prefix use with the name of the crate.

[withoutboats]

What if the first part of use always had to be a crate name? So even in the case of importing internal modules you would’ve to prefix use with the name of the crate.

We can’t do this because you can have a crate that has dependency on another crate with the same name. For example, the binary cargo depends on the library called cargo. However, what we’ve decided to propose for the RFC is something essentially the same, except that the current crate is always called crate instead of its name. So like:

use std::io;
use crate::module;
::std::io 
::crate::module

This was neither @aturon’s nor my favorite solutions (our favorite solutions differed significantly), but as we sat with it we realized it had a lot of pretty clear benefits and was sort of the pragmatic choice. More justifications forthcoming in the RFC in a few days.

[RalfJung]

use std::io;
use crate::module;
::std::io 
::crate::module

So no [std]::io? Well I guess I’m looking forward to the further justification then.^^

[johncf]

To my eyes, pub in pub export seems redundant. In my opinion,

export path::name;

should export name as public by default (without an alternate way of writing that), and the more restricted access levels should be specified using a parameter to export like:

export(self) path::name;

[steven099]

Yeah, the connotations of “export” are suggestive, but potentially misleading. “Export” implies you’re taking something from inside the module and making it available outside the module. Having to say pub export to make it actually available outside the module seems like saying “yeah, but now really do it”. On the other hand, saying export(self) seems like saying “export, but never mind”. I suppose pub(restricted) already suffers from this problem in general, though.

What export is actually doing is bringing a name into the module as an item (as opposed to just useing it in the module). Items declared in the module are already “exported” (even if they’re not public). So it seems like what you’re really doing is importing, though if Rust had both import and use, there’d be no end of confusion about which to use when. “Include” also has more appropriate connotations, but again, people would get confused about when to include and when to use.

“Export” might be close enough, though, as long as you liberally interpret “inside” the module to include submodules, and “outside” the module to… also include submodules. I almost want to suggest—as an alternative to mean bringing a name into the module as an item rather than just useing it—a term which came up near the beginning of this saga: mount. Having mount be visible only to submodules but pub mount be visible publicly would actually make sense, and there’s no risk that people will associate it with the equivalent of use in another language.

Whether you’d be able to get away with using module relative rather than absolute paths is unclear though. The one benefit of export implying taking from inside the module (when it really needs to take from anywhere but inside the module) is that it at least makes the fact that it takes module relative paths self evident.

Edit: I’ve talked myself out of the following train of thought, but I’ve left it in for posterity:

Another term that has very close to the right meaning is “alias”. It would also justify using module relative paths instead of absolute paths. The only thing that might be confusing is that people might expect it to be

[ visibility ] "alias" ident "=" path ";"

whereas here we would want it to be

[ visibility ] "alias" path [ "as" ident ] ";"

The other issue I can foresee is people not expecting alias to be an item, and thus not seeing the distinction from use. In particular, the ident in use path as ident would also be referred to as an alias, potentially confusing people. Whereas mount doesn’t have this problem, since telling people you can use something under an alias or mount it under an alias is relatively easy to understand. Also, “mount” has a very clear connotation of installing something for display, so it would never be mistaken as not being an item.

[withoutboats]

Not to tease things, but the final RFC is closer to that than the current syntax.

Planning to work on the RFC through the weekend and post it on Monday morning.