Revisiting Rust’s modules, part 2

[eridius]

I’m not going to comment about the rest of this proposal, but I just want to say that I rather dislike the fact that this gets rid of private modules.

[KalitaAlexey]

I do really like having from ... import ..., but it seems most people don’t like it.

If I understand it right, we want to make universal paths which could be used in any context. I’d suggest use the following:

Path related to the current module - just path, i.e., solve::Program

Path related to the current crate - a path starting with ::crate::, i.e., ::crate::a::solve::Program

Path related to the extern crate - a path starting with ::extern::crate::, i.e., ::extern::crate::std::io::Error

It can be used for use:

use solve::Program;
use ::crate::a::solve::Solver;
use ::extern::crate::std::io::Error;

It can be used in other contexts:

fn solve_program(program: solve::Program) -> ::extern::crate::std::io::Error {
    ::crate::a::solve::Solver.solve(program)
}

I understand that uses related to either the current crate or to an extern crate are verbose, but it is very easy to read and it makes clear distinction. The only problem I can think of is that people might forget if they need to write :: or not.

Some people could try to write ::solve::Program or crate::std::io::Error.

The I can think of two solutions:

• Add :: to path related to the current module like ::solve::Program and then it is consistent.
• Remove :: from other paths like crate::a::solve::Solver and extern::crate::std::io::Error.

The first solution breaks all code written so it’s only for exhaustiveness.

I’m in favor of the second variant, but it requires us to forbid making modules with names extern and crate. I don’t know if it is allowed now.

[rpjohnst]

I wonder how much boilerplate and/or confusion we’d end up with if implicit modules were private by default rather than pub(crate). I suspect it would work out okay, since we’d still be getting rid of the distinction between pub mod and pub use, and the from/use stuff would improve on beginners’s initial mental models.

The experience would go from (with the current proposal) “try to write some_dir::some_file::some_fn; get an error that some_fn is private” to (with default-private modules) “try to write some_dir::some_file::some_fn; get an error that some_file is private.” Which is still better than getting an error that some_file doesn’t exist.

[zackw]

I don’t have enough experience with complicated uses of the module system to have an informed opinion about most of this. But I want to put in a word of support for two things.

1. I feel very strongly that there should be an explicit list of every source file that’s going to be linked into each crate of a project, somewhere; adding a file to the directory hierarchy should have no effect by itself. I don’t particularly care where the list(s) go; they just need to exist.

   The reason for this is, if I happen to be working on the files in foo/bar/ and I need to run a quick experiment of some sort, I want to be able to create a self-contained program test.rs right there in that directory, compile it, run it, and then forget about it until I am ready to commit stuff to version control. If the top-level build were to pick that file up, that would be bad.

   (If things get messy, there could easily be test.rs, test2.rs, argh.rs, barf.rs, and bug-report.rs piling up.)

2. I would like mod foo { contents } or equivalent functionality to stay, please. More generally, I would like it always to be possible to mechanically transform any crate, however it’s spread over the filesystem, into a single .rs file such that rustc --appropriate --switches single-file.rs gives you the exact same compiled code you would have gotten from the original (modulo debugging information).

   This is invaluable for bug reporting; if I need to tell a library author or the compiler team or whoever to look at a test case, that goes so much more smoothly if I can say “here is the file, compile it” than “split this up as follows and then compile it”.

[withoutboats]

The reason for this is, if I happen to be working on the files in foo/bar/ and I need to run a quick experiment of some sort, I want to be able to create a self-contained program test.rs right there in that directory, compile it, run it, and then forget about it until I am ready to commit stuff to version control. If the top-level build were to pick that file up, that would be bad.

Do you use cargo? Maybe I’m ignorant but I don’t know how you could do this with cargo.

[zackw]

Addendum: I may in fact be confused about the way the current system works, but is it possible that part of the problem with mod foo; is that the “explicit list of every source file that’s going to be linked into a crate” isn’t all in one place? In a C program, that list is all together in the Makefile, but in Rust I think it winds up being spread all over the module hierarchy.

[zackw]

Do you use cargo? Maybe I’m ignorant but I don’t know how you could [have a one-off program get picked up as part of a larger crate build just because it happens to be inside the crate’s source tree] with cargo.

I have used cargo, but not for anything sufficiently complicated to know one way or another whether this could happen with the current system. But that’s not the point — the point is that some of the “take the module hierarchy from the file system” proposals at least sound like this could happen with them.

[FlorentBecker]

I like the idea of a universal path syntax for use with use, but use stuff from crate is surely nice to read. How about making module::submodule::Item from crate the way to write paths to items in external crates, including in compiler messages. Parsing might be a problem, or brackets would be mandatory, which may be a way to discourage using these without use.

In code:

use stuff; // local
use ::other::stuff; // absolute, within the crate
use some::more::stuff from some_crate; // from external crate some_crate

fn vec_from_thin_air() -> Vec from vec { // do we want to allow this? Can we parse it?
fn vec_from_thin_air() -> (Vec from vec) { // with mandatory brackets

In compiler messages:

type (module::SomeType<'a> from krate) does not live long enough […]

[withoutboats]

Sorry, let me be clear: I don’t know how you could build a one-off crate from inside the src directory with cargo today. What you’re describing is not a workflow that we support out of the box AFAIK. Your hypothetical seems premised on you using a workflow that our tools already don’t support.

Do you actually create a file called test.rs that you then build, today? If so, how do you do it?

[briansmith]

More awkward for the implementer, or for the user? I am talking about what’s the better UX for the user, which is more important. Regardless, I don’t see how it could be significantly complicated to implement given the explicit nature of Rust (dependencies are explicitly listed in Cargo.toml, in particular) since an IDE needs to support “go to symbol” and similar functionality anyway.

[zackw]

Sorry, let me be clear: I don’t know how you could build a one-off crate from inside the src directory with cargo today.

Oh, okay, that makes a lot more sense.

I wanted you to imagine that cargo or similar was in use for the “top-level build” but the one-off program was compiled with a direct manual invocation of rustc. In the hypothetical, I have a shell window open with current working directory inside the source tree, and anything more than vi test.rs … rustc test.rs … ./test is too much ceremony. And cd ../../.. or opening another shell window is Right Out, because then I will forget where I was and what I was doing.

[phaylon]

It also reads well enough for traits, though it gets a bit odd with nesting:

impl fmt::Debug from std for MyType { .. }
impl MyTrait for boxed::Box<error::Error from std> from std { .. }

since the name of the symbol boxed::Box can end up quite far apart from its source std.

I would also think this could make it tough to have macro_rules! understand and construct paths.

[FlorentBecker]

It also reads well enough for traits, though it gets a bit odd with nesting:

impl fmt::Debug from std for MyType { .. }
impl MyTrait for boxed::Box<error::Error from std> from std { .. }

I would nest the second example as the following, which reads slightly better:

impl MyTrait for (boxed::Box from std)<error::Error from std> { … }

On the other hand, as I said, awkwardness in this case is not a problem in code, as it nudges people towards use, but it might be good to have the compiler say something like:

The trait (fmt::Debug from std) is not implemented for the type boxed::Box<error::Error>
    using:
      - boxed from std
      - error from std

Rather than the less nice

The trait (fmt::Debug from std) is not implemented for the type (boxed::Box from std)<error::Error from std>

[jntrnr]

More awkward for the user. Consider from/use and use/from.

In from/use, typing “from” we have context we want a module next. Then “use” now the next completion we can down select to a list of exports of that module. No backtracking, and one step leads naturally to the next.

In use/from, “use” then the completion list is all possible exports from all modules, a potentially large list, then “from” then a completion we can potentially automatically filled out if we know it, though things like wild cards might make this tricky.

Of the two using the module to give you a list of exports lets you explore more easily, and has a generally nicer flow for most folks.

[newpavlov]

I think this proposal is definitely much better than the previous one! So my part of feedback and some ideas (a bit random for now):

• pub use is often referenced in terms of “exporting” something, so why not to call it just that? So we’ll write export foo; for exporting whole module to outside world, and export foo::{bar, zoo}; for exporting elements or sub-modules, and pub use will only be used for sharing inside crate. (of course export'ed items will be visible inside crate too) This way facade pattern can be expressed naturally using multi-line approach without much boilerplate and problem of “automatic promotion of key items” will be solved too. I think it certainly will help with teaching and intuitiveness of the module system. And maybe then even rename from/use to from/import for name coherence and to please Python crowd.
• I think from/use will be better compared to use/from, as it’s represents top-down approach more intuitive for hierarchies. And in general I like removal of extern crate in favour of from/use and I think it has a nice readability compared to proposals based on use ::cratename... or use extern::cratename::.... Although we need to consider public re-exports and how they will look.
• It would be good to allow from crate use self;, so items could be referenced as crate::foo. Or just copy Python and use from/import and import crate;. This way re-export will look as import crate; export crate;. Alternative would be pub import crate;, but I don’t like it much.
• I personally feel explicit use's (or mods) for bringing module into the crate scope is a must have. One of the big negative points in the previous proposal was implicitness (i.e. taking module structure from file system), which I strongly dislike and which I think will have an undesired impact on code discoverability and readability. (yes, I am generally not using IDEs) Also warnings can significantly reduce value of mentioned potential problems. So in some sense use will replace current mod for files with only impl blocks. But at this point why not just leave mod as is?
• Continuing on x/x.rs problem I think we can allow to write mod foo::bar::zoo for adding foo/bar/zoo.rs into crate scope without requiring mod.rses in the foo and bar directories and allow to write use zoo::item;, so we’ll forget exact path to zoo as an implementation detail.
• About absolute/relative paths I am more or less indifferent, with slight favour of keeping absolute path to reduce breakage.
• About pub(crate) by default, I personally don’t have crates large enough to have need for finer privacy granularity, so I can’t judge, but maybe need for it is an indication for time to split project into several crates?

[edmccard]

anything more than vi test.rs … rustc test.rs … ./test is too much ceremony. And cd …/…/… or opening another shell window is Right Out,

Using only a single shell or editor window (unless you have a tiny monitor?) seems like a pretty niche workflow but anyway, for running a one-off test program, what I do is vim ../../examples/xx_test.rs then cargo run --example xx_test which is no more ceremony than vi test.rs … rustc test.rs … ./test. I also have a .gitignore rule that ignores files starting with xx.

That said, would it work for you if the “files become modules” rule respected .gitignore? That’s something that I think any version of this module proposal should take into account.

[zackw]

Using only a single shell or editor window (unless you have a tiny monitor?) seems like a pretty niche workflow

Instead, please imagine that there are 10+ shell and editor windows, each associated with different subdirectories or perhaps even different projects. Some of them may be ssh’d into remote machines. Keeping to a strict association of windows with specific locations is the only way I don’t get hopelessly lost.

vim ../../examples/xx_test.rs then cargo run --example xx_test

It’s important for several other reasons that the test program’s source code be in the same directory as the other stuff: It is often derived from the code in that directory, so I’m copying and pasting between the two and I want the editor to look in that directory, not ../../examples, when I use :e, :r or equivalent (note: I don’t actually remember what vim does here); it may, when run, process code or data in that directory; it might not even be intended to run at all, just to be a bunch of notes-to-self associated with the code in that directory, but I still want to use the .rs extension for syntax highlighting convenience.

would it work for you if the “files become modules” rule respected .gitignore? That’s something that I think any version of this module proposal should take into account.

No, that would not be sufficient. The most important reason is that these files do get checked in under some circumstances, and may persist for quite some time. Another reason is that there may not be a version control system for something, or not one that the toolchain understands (does anyone here remember ClearCase?)

I don’t want the behavior of the compiler ever to depend on the existence or state of version control. That may under some circumstances be reasonable behavior for cargo, but not for rustc.

[dhardy]

At risk of getting heavily spammed… I’m still not sure what this proposal solves.

Yes, removing extern ... might make sense; I’m unsure myself. Some new privacy rule like pub(mod) is necessary.

And some IDE functionality to assist writing use ... declarations would help; that would probably be a better solution for beginners than switching to relative paths.

[RalfJung]

While we are happily bikeshedding about variants of this in half a dozen threads, there is one thing from the first proposal which is absent in almost all other proposals, and which at least I personally liked a lot: Having files be anonymous modules. By this I mean that items in “bar/memory.rs” and “bar/cast.rs” all end up in the “bar” module directly, rather than in two separate submodules – while both files can still also have private items that are not accessible even to the other file. That is entirely orthogonal to the discussions around from ... use and path syntax and whatnot, so having these discussions separately makes perfect sense to me. However, it is not clear to me whether “files are anonymous modules” is tabled for now because there was significant pushback (was there?), or whether it is missing here because this is a separate discussion we are going to have after reaching some form of consensus on what is currently being discussed.

[newpavlov]

Perhaps you did not read my proposal carefully or maybe my wording was a bit unclear, but use(inline) (or mod(inline) if it will be decided to keep this keyword) expresses exactly your anonymous modules. In my personal opinion it’s a dangerous tool, due to its inherent implicitness and it must be used only if it’s possible to unambiguously infer from which file exported item is originated looking only on file names.