Revisiting modules, take 3

[rpjohnst]

The only way to do that is to completely throw backwards compatibility out the window, which seems a far bigger problem to me. Perhaps allowing the new universal paths in use could be made to work, but forcing a new prefix onto every intra-crate use path is a terrible idea.

[withoutboats]

However, I don’t like the fact that in this proposal, use module::Type and const x: module::Type = … do not use the same rules to resolve module::Type.

With that in mind, “fixing” the issue by eliminating pub use and adding yet another keyword seems like a step backwards. We already learn pub use as “works like use, but also reexports the name”.

Let’s please introduce a new universally clear, nicely separated form, that puts external and internal items in clearly separated namespaces, and then use that exact form in the use statement, please. It’s incredibly intuitive that use uses the same paths that work everywhere else; please don’t break that in the process of improving it.

To restate these objections as I understand them:

• Its not great that use uses different paths than those inside the module.
• Separating use and export into two different keywords is worse than just having pub use work differently than use.
• Imports/exports should just use the absolute path syntax.

I think all of these objections are reasonable. I chose pretty opinionated default settings, and I didn’t clearly articulate the motivation for them.

I’ve experimented with applying this and the previous proposal to existing crates (mainly futures and chalk) and made these observations:

• Absolute paths are rare inside modules; inside modules you almost always want self:: paths.
• Import statements (use) are mostly from the crate root (crate::); many are also from extern crates (extern::); a small number are from this module or the parent (self:: and super::).
• Export statement (pub use), in contrast, are mostly from this module (self::).

Based on this learning, I made these conclusions:

• Import statements should have the crate root as the default, and it should be easy (and hopefully self-explanatory) to import from other crates as well. It should be possible to import from anywhere.
• Export statements should have this module as the default. It should be possible to import from anywhere.
• It should be possible to use a path from anywhere inside a module, but it isn’t super important that it be the most abbreviated form.

And from these conclusions I think the design decisions follow:

The notion of path prefixes

Absolute paths have a prefix, which is one of:

• self, refering to this module
• super, refering to the parent module
• crate, refering to the crate root
• extern::<crate>, refering to a dependency

An absolute path, outside of an import/export statement, uses the syntax <prefix>::<path>.

use and export vs use and pub use

There are two motivations here:

• I wanted the default prefix for import to be different than the default prefix for export
• I wanted use to not introduce items visible outside of this module, even in submodules.

Changing what it means to use foo::Bar by adding a visibility modifier seems very surprising to me. That use foo::Bar means to search from the crate root but pub use foo::Bar means to search from self is troubling.

Of course this is an argument against having different defaults here. Ralf made the case that all paths should have the same default prefix, whereas Josh made the case that import/export statement paths should not have a prefix specified at all. But I think, with two different keywords, having different default settings will be reasonable to learn.

The from syntax

There’s also objection to the from syntax as the way to change the prefix of a path in import/export statements. I want to point out that its actually shorter in the current form of this proposal for most cases. That is:

from std use io;
// vs
use extern::std::io;

In addition to being shorter, I think the first one is more intuitive.

Because absolute paths inside of modules are rare, I think users will learn the extern::std::io syntax later on, and will at first be using the from std. I would liken this to type ascription and turbofish. At first, I know that I can do:

let x: Vec<_> = iterator.collect();

But then I have a situation like this, where I don’t want to create a temporary:

s1.parse()? + s2.parse()?

At this point, I learn about the turbofish syntax:

s1.parse::<u32>? + s2.parse::<u32>()?

I would even say that they’re conceptually analogous - creating a temporary is like performing an import, whereas using an absolute path inside your code is like needing to inject the type into the middle of an expression with turbofish.

I might prefer supporting from as the only syntax for absolute paths also, though I don’t know how I (or anyone else) would feel about that:

let x = from std vec::Vec<i32> = vec![];

[withoutboats]

I personally do not think “use is an item” is particularly hard to learn

I just wanted to drop one more thing because crate as a keyword came up repeatedly here: I do not find it obvious at all that this means current crate.

I want to contrast these two statements to try to explain why I think “Use is an item” is a hard idea, and “crate means this crate” is an easy idea. The difference between them is what I’m going to call “generative implications.”

By generative implications, I mean you have this simple idea, but it has implications that ripple throughout the system leading to complex and non-obvious consequences. Computer scientists love generative implications, and I think PL people even more so - look, with simple ideas A, B, and C we have turing completeness! And generative implications in semantics are what give the experience of programming a lot of its richness of experience, you always feel like you’re learning more things about how to use the language you thought you already knew.

“Use is an item” has generative implications; things like how if you import a name into the root module, you can import it anywhere without any prefixes are not obvious, but once explained clearly follow from that simple concept

“Crate means this crate” does not have generative implications. Its a piece of syntax which may have two meanings that are obvious to someone, and so they ask “does crate::foo mean ‘foo’ from this crate or the crate foo?” and they get the answer - "its the first one, the crate foo would be written extern::foo". There’s no deeper insights to learn later about what this concept implies.

My claim is that in nonsemantic systems of the language, like name resolution, we should be wary of concepts with generative implications, because they are surprising to users and can, in the worst case, lead users to develop inaccurate mental models because they have misinterpreted some of those implications.

[josh]

Ignoring the from here, the rest of this version of the proposal looks perfect to me. And thank you for taking feedback into account.

I’m encouraged by the shift towards increased consistency.

[le-jzr]

To restate these objections as I understand them:

• Separating use and export into two different keywords is worse than just having pub use work differently than use.

Not intrinsically, but backwards compatibility prevents us from simply removing the old syntax. pub use will stay in the language whatever the conclusion, and adding almost identical construct that works from different root sounds like more trouble than it’s worth, learnability-wise.

extern::<crate>, refering to a dependency

Something about that syntax doesn’t feel right to me. Local crate is its own root, but external crates all share a single root. I can’t quite rationalize it, but if I had OCD, it would be in overdrive here. Plus it’s awkward to write.

How would you feel about figuring out something where the external crate is syntactically the first component of the path? @my_crate::path has been suggested, but might not fit well into the parser. Possibly it could be [my_crate]::path? Can anyone think of other possible variants that parse and don’t look like a roadkill?

But I think, with two different keywords, having different default settings will be reasonable to learn.

I think your basic premise here is that the current “path confusion”, which has been discussed as a significant problem, is actually not a problem at all. Which is a perfectly valid opinion, but the fact that you don’t even acknowledge path confusion as an established phenomenon leaves me wary of your conclusions.

There’s also objection to the from syntax as the way to change the prefix of a path in import/export statements. I want to point out that its actually shorter in the current form of this proposal for most cases.

My version is even better:

 use [std]::io;

from is decent as a workaround for the awkwardness of the proposed syntax, but I think that with properly designed base form, there shouldn’t be any incentive to invent sugar for it.

[withoutboats]

I think your basic premise here is that the current “path confusion”, which has been discussed as a significant problem, is actually not a problem at all. Which is a perfectly valid opinion, but the fact that you don’t even acknowledge path confusion as an established phenomenon leaves me wary of your conclusions.

I don’t think “path confusion” is as simple as it is made out to be. That is, I think the underlying problem is not that import statements take an absolute path, and local statements take a relative path. In the previous thread, @theduke listed several minstream languages which use absolute paths for their import statements. I would add Ruby to this as well, though its UX is a bit worse because its require-based.

Our problem is a bit different, and I think its got more to do with the shift from a single-file project to a multi-file project. I think in a single-file project, you get bad hints (from the everything is an item situation) that lead you down the wrong road.

For example, you might think that once you’ve added a dependency, you can use it everywhere. But the reason you can use it your main.rs file is that extern crate foo; is an item, which adds “foo” to the namespace of the root. That’s not an obvious fact to many users. Solution: we make all imports from external crates act the same way.

Another problem is that you write code like this once you add a submodule to your project:

mod foo;
use foo::Bar;

Now you have both foo and Bar in scope. You might think that because you declared mod foo;, you can now use from foo. But this is only true in the root module, in other modules you have to say use self::foo::.

A way to mitigate this is to only have imports and not mod statements. If you don’t say mod foo, maybe you don’t build a mental model that you’ll be able to say use foo because you have that special “mod import” that you have to do when you add files.

I don’t think this problem is entirely mitigated, and I would like for use foo, where foo is a submodule, to work in every module. I’ve had trouble finding a solution though, because of the problems that fallback creates with re-exports that pcwalton mentioned in the earlier thread.

---

However, I think path confusion is also mitigated to some extent by the from syntax. That is, lets say your imports look like this:

from std use collections::HashMap;
from std use fmt;
from std use path::{PathBuf, Path};

from semver use Version;
from serde  use ser;

use core::{Dependency, PackageId, Summary, SourceId, PackageIdSpec};
use core::WorkspaceConfig;

(These are the imports in cargo’s core/manifest.rs module.)

My belief is that by making from <crate> a separate syntactic unit, users are hinted that the use core statements have had a from component elided. This makes it easier to internalize that bare use statements have the from crate statement elided on them.

Similarly, I think that the from syntax also makes it easier to understand that use and export have different elided prefixes. “use = from crate, export = from self.”

[RalfJung]

Separating use and export into two different keywords is worse than just having pub use work differently than use.

As expressed in another thread, I would not agree with this. Having use and pub use be fundamentally different seems very confusing to me.

I’ve experimented with applying this and the previous proposal to existing crates (mainly futures and chalk) and made these observations:

Absolute paths are rare inside modules; inside modules you almost always want self:: paths. Import statements (use) are mostly from the crate root (crate::); many are also from extern crates (extern::); a small number are from this module or the parent (self:: and super::). Export statement (pub use), in contrast, are mostly from this module (self::).

Based on this learning, I made these conclusions:

Import statements should have the crate root as the default, and it should be easy (and hopefully self-explanatory) to import from other crates as well. It should be possible to import from anywhere. Export statements should have this module as the default. It should be possible to import from anywhere. It should be possible to use a path from anywhere inside a module, but it isn’t super important that it be the most abbreviated form.

And from these conclusions I think the design decisions follow

I agree that having different interpretations for paths in use and elsewhere probably leads to slightly shorter code. However, I think the confusion and mental complexity introduced by having to remember, when reading a path, how exactly this depends on the context is way worse than having to write a few more letters. So, even though use is usually going to need an absolute path, I feel strongly that it should (when no explicit root is given) still be relative, because consistency with the rest of the file outweighs character count.

And I don’t think this is speculation. We have such a system right now, and experience shows people are confused by it. @aturon mentioned that in his 2nd proposal. @le-jzr already expressed his surprise that you did not even acknowledge the problem of “path confusion”.

You later mention the problem of people building the wrong mental model. I think the “path confusion” is one of these points where people build the wrong mental model. When they start experimenting a little, they will notice that this actually works:

struct Type;
mod foo {
  use Type;

  fn bar(t: Type) { ... }
}

The reason this will work is that they are in a small crate, and this is happening at the crate root. Some time later, they try doing the same thing deeper in the module hierarchy, and they will be extremely surprised by this suddenly not working any more.

So, I still strongly object the idea that paths in use (or export) should start anywhere but in the local module. Saving some characters is not good enough an argument.

---

Re: “generative implications”: I am not strongly attached to "use is an item". I think it is beautiful, and it is one of the things in Rust that struck me as a nice little bonus compared to other languages – no more distinction between items declared locally and items imported elsewhere when doing lookup, what a great idea! But I can accept people telling me that being more like other languages here makes Rust easier to learn and is thus overall advantageous. The objections I raised against export are not motivated by the desire to re-unify use and export, but rather by the desire to hopefully make this feature better understood.

[withoutboats]

So, I still strongly object the idea that paths in use (or export) should start anywhere but in the local module. Saving some characters is not good enough an argument.

I attempted to articulate (though i think in parallel to you drafting your response) that “path confusion” isn’t about imports working differently from local paths - you sort of hit on the same thing, that the problem is how they work the same as local paths in the crate root. I think fixing this for external dependencies significantly mitigates the problem, though I’m unhappy it still exists to some extent with modules.

In general, I am sympathetic to “being easier to understand is worth being longer,” but here are some very rough numbers, generated using grep:

• chalk - 122 non-public use statements, of which 4 are self:: prefixed
• cargo - 658 non-public use statements, 18 self:: prefixed
• diesel/diesel: 669 / 37
• futures - 356 / 4
• hyper - 508 / 14

These numbers seem just overwhelming to me that self:: would be the wrong default prefix for import statements.

[RalfJung]

I attempted to articulate (though i think in parallel to you drafting your response)

Yeah sorry, I didn’t see that post while writing mine.

I don’t think this problem is entirely mitigated, and I would like for use foo, where foo is a submodule, to work in every module. I’ve had trouble finding a solution though, because of the problems that fallback creates with re-exports that pcwalton mentioned in the earlier thread.

Fallback to the parent module is not currently happening, is it? So wouldn’t an alternative be to not keep not having fallback? (I wasn’t even aware your proposal involves such fallback; must have missed this when you wrote it.)

Another option may be to require things to be more sorted: When there’s an inline module (I assume fallback doesn’t work for file modules either way, that would just be scary), then you cannot use things inside that module before the module. That should avoid the problematic cycles, shouldn’t it?

Having use import stuff for the entire rest of the file, not just for this module, sounds like a nice idea indeed. However, even then you can still create cycles with from self use ..., right?

My belief is that by making from <crate> a separate syntactic unit, users are hinted that the use core statements have had a from component elided. This makes it easier to internalize that bare use statements have the from crate statement elided on them.

That is an interesting take on this. I have not thought of this before – rather than making from use sugar for use, you are making use sugar for from from use. This is certainly a much better explanation for “why do things stop working when I move away from the crate root” than “well, paths in use are different”. I will have to think about this.

Similarly, I think that the from syntax also makes it easier to understand that use and export have different elided prefixes. “use = from crate, export = from self.”

I think that’s still rather subtle.

In general, I am sympathetic to “being easier to understand is worth being longer,” but here are some very rough numbers, generated using grep:

chalk - 122 non-public use statements, of which 4 are self:: prefixed cargo - 658 non-public use statements, 18 self:: prefixed diesel/diesel: 669 / 37 futures - 356 / 4 hyper - 508 / 14

These numbers seem just overwhelming to me that self:: would be the wrong default prefix for import statements.

If we get better grouping of imports (use extern::std::{ /* multiple lines of imports, with nested curly braces, in here */};), that shouldn’t end up being much more to type than the from ... use you propose.

I don’t even strongly mind the from sugar; it should be sufficiently clear that this changes the root with respect to which the immediately following use is interpreted. I’d be fine with linting people to use from std use rather than use extern::std:: – using a style lint to provide better ecosystem consistency, just like we do for snake_case vs. CamelCase naming.

I think the only point we really disagree on is how to interpret use without from. So this is about use foo vs. from crate use foo – we are talking about 11 characters per file (assuming we can do from crate use foo, bar, baz):

from std use
  collections::HashMap,
  fmt,
  path::{PathBuf, Path};

from semver use Version;
from serde  use ser;

from crate use // or maybe "from this use? "from this crate use"? ;)
  core::{Dependency, PackageId, Summary, SourceId, PackageIdSpec},
  core::WorkspaceConfig;

[josh]

use [std]::io;

This syntax is growing on me the more I stare at it.

[RalfJung]

use [std]::io;

This syntax is growing on me the more I stare at it.

I find it gets even more appealing if you compare it with crate-local absolute paths:

use [std]::collections::HashMap;
use [std]::fmt;
use [std]::path::{PathBuf, Path};

use [semver]::Version;
use [serde]::ser;

use ::core::{Dependency, PackageId, Summary, SourceId, PackageIdSpec};
use ::core::WorkspaceConfig;

Or with full use of nested curly braces:

use [std]::{
  collections::HashMap,
  fmt;
  path::{PathBuf, Path}
};

use [semver]::Version;
use [serde]::ser;

use ::core::{Dependency, PackageId, Summary, SourceId, PackageIdSpec, WorkspaceConfig};

Yeah, I think I do prefer this over extern:: and crate::, and it also seems (at least) on par with from, while also being shorter and not providing two different ways to express the same thing.

[withoutboats]

That syntax seems to have very little self-explanatory power, which is one of the chief advantages of from in my opinion. I have to admit I am very unenthusiastic about it.

[RalfJung]

That syntax seems to have very little self-explanatory power, which is one of the chief advantages of from in my opinion. I have to admit I am very unenthusiastic about it.

Just like “Which crate does crate::foo refer to”, I think you can learn this once and be done. Admittedly, the syntax here is impossible to google. OTOH, pretty much every example will have to import something from std, so this will be explained very early on everywhere.

But yes, I admit from std use foo is much more self-explaining than use [std]::foo.

[josh]

from doesn’t work consistently for absolute paths elsewhere, though, and extern:: is really verbose there. [cratename]::module::name is a syntax that could work absolutely everywhere, including both use statements and code.

[arielb1]

I’ve experimented with applying this and the previous proposal to existing crates (mainly futures and chalk) and made these observations:

Turns out this differs across crates:

• In the rustc repo, there are 10545 uses, of them 500 (4.7%) are module-relative, at least 6459 (61.3%) are cross-crate, and less than 3848 (34.0%) are crate-relative.
• In “small scripts”, it’s even worse - consider that 546 of Cargo’s 762 (71.5%) uses are cross-crate.

This means that users of these sorts of crates have differing experiences on the effects of making imports relative. futures and chalk aren’t exemplary of all crates.

Statistics were gathered using this command and hand-inspecting the results:

find src -name \*.rs -print0 | xargs -0 cat | sed -rn 's@^use ([^:;]+).*$@\1@p'

[arielb1]

That syntax seems to have very little self-explanatory power, which is one of the chief advantages of from in my opinion. I have to admit I am very unenthusiastic about it.

I think use [std]::fmt; is pretty self-explanatory. It’s also:

• reasonably concise
• not ugly
• can be used in item_path_str paths (in error messages) - [std]::rc::Rc<&mut [std]::boxed::Box<&i32>>.

Users can quickly see that “a crate you are importing from must appear between brackets”. It also meshes well with absolute path syntax - ::rc::Rc in std is [std]::rc::Rc out of it.

[elahn]

I’m loving use [std]::fmt; and combined with :: for absolute paths everywhere aka relative paths everywhere, doesn’t that completely solve “path confusion” aka “works in main/lib.rs but not elsewhere?”

struct Type;
mod foo {
  use ::Type;  // must use absolute path or super::Type

  fn bar(t: Type) { ... }
}

The more I look at ::, the less ugly it looks and I think after a while it would be “normal.” And it’s a small price to pay for consistent paths in use, export and regular code.

[josh]

[crate] even seems rather evocative: with square brackets, it looks like a long box, or in other words, a crate.

[le-jzr]

I think export would be worth it if leading :: stays as the root of current crate. Then the migration story is very simple: deprecate absolute paths in use that don’t start with :: (using a simple lint), and add export with relative paths.

That way, the semantics of use paths don’t need to change at all – the confusing part just gets phased out over time as people fix the warning, or just run updated cargo fmt on their crates. The cost of two characters per import is well outweighed by the uniformity of paths everywhere.

[rpjohnst]

I also prefer [crate]:: to extern::crate::. It feels like a lighter-weight change to the current system, as it is much less invasive into how paths work. I especially appreciate the symmetry between [some_other_crate]::item and ::some_item_in_this_crate, as the :: syntax is already familiar to C++ programmers.