Revisiting Rust's modules

[withoutboats]

If you want to comment out an entire directory you would just have to rename the directory to any name that would be an invalid ident.

You’re communicating in an uncharitable way.

[withoutboats]

a) this will make the layout for “simple” projects more complex.

I have the same question I had for @lambda: why is it important to you that these be separate modules - in the sense that their names appear in the namespace hierarchy?

[phaylon]

An #![ignore] inside would have to not require the file to compile. It would also still be less convenient than a facility in the parent module. Those are a lot easier to track when you just comment all of them out, and then uncomment one at a time while you fix them up during refactoring.

I guess something like #![no_load(..)] might work on the parent. Though I have to say I’d still be happier with the current way, since I don’t even have to find out what I’d have to put there, I just comment them all out.

I like that workflow because it kind of trained me to get better at structuring my projects.

[rpjohnst]

I don’t mean to be uncharitable; it’s just that renaming to an invalid ident is not a very obvious solution! It didn’t occur to me and I doubt it will occur to beginners for whom the directory proposal is supposed to be easier.

So instead of accusing each other of poor communication, let’s focus on communicating better.

[rpjohnst]

I’m not either of them, but to me it seems that’s not really a great question. I would take it as a given that there are plenty of “simple” projects that do want actual namespacing separation as described. Otherwise why even bring it up? I know I have a couple projects that work that way.

[lambda]

But why do you need them to be modules, if you can split them into separate files & privacy scopes without making them modules?

You may want to have public items that are not automatically in the namespace of every other piece of code in the project. For instance, sometimes you have local Error types, that you don’t want to have to rename redundantly with the name of the file they are scoped in, but you do need some of the modules to have machinery for converting them between each other.

Also, the example I gave was for a simple executable, but the same would apply for a simple library. You may want the module structure to be part of your API. For instance, you have some of the most common functionality in the crate root, and then a few small modules for various helpers for various tasks, that you want to group together in separate modules so they don’t clutter up the main docs/namespace/autcompletion space.

[withoutboats]

So instead of accusing each other of poor communication, let’s focus on communicating better.

How could I try to communicate better with you without telling you how I feel you are communicating?

I’m not either of them, but to me it seems that’s not really a great question.

How can I understand how to handle users’ needs without asking what they are?

---

A way you could communicate more charitably is expressing uncertainty and phrasing your comments as questions. Both of your previous replies to me would have been better phrased as questions.

[withoutboats]

This certainly favors cetain code patterns that are different from today, but today’s system also puts burden on certain patterns - facading, for example, is a lot of boilerplate.

There’s a taxonomy which divides module structures into “wide modules” and “deep modules.” In wide modules, your public module has a bunch of private submodules, and you re-export the types in the private submodules in that public parent. futures::future is a good example of that. In deep modules, you instead export things in a matter that very closely mirrors the exact canonical path, so even publicly you have “a module per file.”

The system we have today favors deep modules over wide modules, but our investigations and personal experiences lead us to believe that wide modules are common and the balance is tipped in the wrong direction right now. It seems impossible to maximize ergonomics for both patterns.

[Letheed]

• regarding path confusion, we can simply decide on a different behavior (rel. vs abs.) whenever Rust 2.0 happens. We can just make all paths relative if that’s easier for everyone. I don’t really care either way. I suspect that this issue is largely a case of negativity bias. That doesn’t mean nothing can be done to improve things though.

  • One idea could be to keep the current defaults but if the default mode doesn’t match the first item in the path hierarchy, try the other mode. e.g.

    use map::foo::Baz;
    
    mod map;
    

    would first look for an extern crate map declaration in the crate root, but if none is found would look for a mod map in the same file.

    Advantages:

    • backward compatible, so we don’t need to wait.
    • simple to use

    Drawbacks:

    • a bit more complex conceptually
    • adding extern crate map; or adding a module can be a breaking change. (Though you just need to use extern crate map as map_crate; or pick another name for your module.)

  • Other idea: remove defaults altogether. Try both absolute and relative resolutions and demand that one and only one matches. If both do, demand the use of either ::map::foo::Baz or self::map::foo::Baz to disambiguate. In the vast majority of cases, that should alleviate the need for ::path and self::

• regarding extern crate, we could add an auto_link field (or whatever the name) in the dependencies section of Cargo.toml (defaulting to false). When set to true, for every declared dependency abc, extern crate abc; is implicitly added to the root file of every lib/bin in the crate. The template Cargo.toml manifests generated by cargo new and cargo new --bin would contain auto_link = true. Beginners could just add dependencies and start coding, no muss no fuss. This would also make things simpler for crates with a single bin/lib. Complex setups can delete the auto_link field and declare their extern crate manually like today if they don’t wish to import everything everywhere.

• regarding foo.rs vs foo/mod.rs I agree this has several problems:

  • it is additional complexity for beginners
  • when adding submodules to a module, I need to move it and rename it.
  • when I have many files open, it isn’t rare that half of them are called mod.rs which is really unpractical to distinguish them.

  We could allow for a third way: foor.rs side by side with the foo/ folder. Submodules, if any, go into the folder. No need to rename, no more bizarre mod.rs files, no more moving.

The current module system is not the easiest but in its favor it is perfectly explicit and as flexible as you want it to be. The proposal trades that for implicitness which some will like but others wont and at the cost of flexibility, which I’m not feeling very happy about. I dislike:

• all my files would actually be one single giant but completely virtual file.
• the _ prefix special casing whereby a special name would have a special meaning.
• I couldn’t include/exclude files manually, just with a line of code. I have to move them or rename them.
• conversely I can’t see what modules are included in a file from within the file itself. When I explore a file, I don’t want to have to grep other files or check all files in a folder. Today if I see mod foo; I know I can just open foo.rs to get what I want.
• tying the internal module hierarchy with the public API hierarchy by making all modules implicitly pub mod. I’d prefer if we kept those separated, even if that means having to list mods and pub uses from time to time.

It feels like gearing the entire module system towards saving people using facades a few minutes of trivial code at the cost of removing everybody else’s expressivity.

Alternatively, if people really want to tie themselves to the file system, we could have a less disruptive change: we remove the mod keyword. Modules are implicitly imported. e.g.

src/
  bar.rs
  lib.rs
  foo/
    baz.rs
  foo.rs

means lib.rs implicitly declares mod bar; mod foo; and foo.rs implicitly declares mod baz; The publicity of the module is declared with an attribute at the top of the module. e.g. #![public(crate)]. The use keyword keeps working exactly as today.

Last words

I honestly like the current module system. I don’t feel saving some keystrokes is worth a complete change of paradigm. I feel like refinement of the current system, like the three changes proposed above would go a long way into making it more palatable already.

[KasMA1990]

I’ve recently spent a lot of energy working with Java and its name spacing system, and I have to say that I really like it; it’s very simple (conceptually) and the language has an abstraction that is a natural fit for one file (a class, with a clear privacy boundary), making the language and the file organisation complement each other in an intuitive way.

I think one of the biggest stumbling blocks for me in the current module system, is that I don’t have a natural abstraction that corresponds to a file, since in my mind, a module is closer to being a name space than a class. Hence I think this proposal is good for many of the same reasons that I like the Java system; it defines that a file should not be a module, because only directories are modules. I think the natural way for me to organize my code would then fall to having one file per type or something similarly Java-ish.

But the question is if that’s a direction we want Rust code to be written in? Organising everything by class is something that has influenced me to write code that is split by types, even when I’ve had to invent a new type, just to be able to organise my code better (AbstractServiceFactory, etc). And I’m guessing many others would do the same in Rust, if that’s the natural way of organising code.

I don’t know if we can simply focus on education as a way around this issue; if many of the people that have problems understanding modules think the same way I do, then maybe it’s just a matter of focusing on telling people how to use modules to get abstractions they want.

There are still some problems that should be fixed regardless though (path confusion is real!), but I’m curious about the effect of basically telling people that files are not modules; what will people then consider files? Because I think many want one file to correspond to some clear unit of code, that is smaller than a module still.

Side note: If Java has taught me anything it’s that things like name spacing can be completely hidden with an IDE, for better and worse.

[bennofs]

I have the same question I had for @lambda: why is it important to you that these be separate modules - in the sense that their names appear in the namespace hierarchy?

It is nice for namespacing purposes. Right now, I can use pretty generic names for types and functions in a module, and then refer to them by module::generic_name. If this proposal is implemented, I’d probably have to name each function as module_generic_name (kind of like it is done in C).

[lookmeat]

I think there’s two things. The first one is that there always is going to be a semantics gap, because directories and filenames can have names that are invalid in the language, and one is intrinsically tied to the OS while the other is independent. I don’t think that’s bad.

I also think that sometimes it makes sense to have per-file module, sometimes it doesn’t, in some weird cases I might want directories that don’t create modules!

I can think of an interesting solution that let us benefit from the best of both worlds (though it doesn’t care about being backwards compatible and the implications yet).

• In a crate directory’s root a cargo.toml file is added that explains how to import external crates. They are exposed as modules on the root (
• The crate directory (defined by containing the cargo definition) has its contents inside the root directory.
• Files and directories whose name does not start with _ create a new module with their name and have their contents inside this module
• Files and directories whose name starts with _ are implicit, and have their contents added to the root directory.
  • (backwards compatibility we’d special conditions for lib.rs and mod.rs).
• The root namespace is always pub(world).
• By default the modules generated by files and directories are pub(crate) (except for the root module). Files and directories prefixed with pub- will be pub(world) and their name ignores the prefix. You can make a directory foo into pub(world) module by naming it pub-foo and a file bar.rs into a public one by naming it pub-file.rs; the module names would remain the same even as the filename changes.
  • Names with _ leading are not creating modules, so it makes no sense, it’s ok as both prefixes are mutually exclusive pub-_baz.rs is an invalid filename. A name _pub-foo.rs could be taken as a valid implicit file, since it doesn’t create a module, but I think we should make it a mistake since it can be interpreted in ambiguous manners.
  • Backwards compatibility we’ll probably have to make things pub(world) by default, and you’ll have to opt out, I like following code convention.
• mod <name> always creates a module within a file. The whole importing file thing is dropped because it’s hard to keep file structure and their contents in-sync (and they are explicit enough on their own, IMHO).

So then we can have a directory that looks like the following:

my_rust_crate
|- cargo.toml // Imports crate crt
|- _lib.rs
|- foo.rs
|- pub-baz.rs
|- pub-bar
    |-_big_function.rs
    |- _other_big_def.rs
    |- _small_stuff.rs
|- _fizz
    |- buzz.rs
    |- _impl.rs

This creates the equivalent namespace:

pub(crate) mod crt { // optionally this module would be inside a pub(crate) mod extern statement.
// Contents of crt crate
// This actually works differently because of crate special rules
// making the crate pub(world) needs to happen at the toml level.
}

// Contents of _lib.rs

pub(crate) mod foo {
// contents of foo.rs
}

pub mod baz {
// contents of baz.rs
}

pub mod bar {
// contents of _big_function.rs
// contents of _other_big_def.rs
// contents of _small_stuff.rs
}

mod buzz {
// contents of _fizz/buzz.rs
// Weird edge case the _fizz directory has its contents dumped, but buzz is a separate file.
}

// contents of _fizz/_impl.rs

Also I’d make all paths work from the current namespace, instead of global, and have a unique way of accessing the absolute path (probably starting them with ::. As long as you don’t overload std (there should be lints preventing that) you can always do use std::... and it’ll do the right thing.

Now some caveats and random thoughts.

• Open question: is it worth it to be limited to OSes that support filenames and directories starting with _, does this make sense?
• There’s also the problem of two mods with the same name. Either a file and directory both with the same name and/or a case where you have foo.rs and pub-foo.rs. The former is not as bad, because we could implicitly declare them both as the same (but to keep surprises low I’d prefer it’d be an error), the latter is very problematic because it’s creating two mods with different visibility rules but the same name.
• Alternative: Could we do something like Go and use capitalization of the first letter to decide if it’s pub(world|crate). This would avoid the problem with two mods with different visibility but the same name. Windows filenames are not case sensitive though.
• It might not be intuitive for new users that files and directories that start with _ are not modules. I think that the solution is to make module names starting with _. When they try to use a file starting with _ as a module, we can realize the probable mistake and educate them of implicit files and directories.
  • This means that we’d have a more complex identifier story for crates, but at least we wouldn’t need magic filenames.
  • Open question: does _.rs make sense? does a directory named _ make sense?
• Backwards compatibility this offers some headaches, and we’ll need exceptions for some of the current conventions ie. lib.rs/mod.rs (even though they aren’t needed any more.
• It might not be intuitive that your library is not exporting all things. It’s also weird that root implicit files don’t need the pub to expose their contents which might lead to newer users using a flat collection of implicit files. What we want is that implicit files are used as the exception instead of the rule, generally new files should store their stuff in their own module to keep things sane in the long-run, we want to make that path easier.
  • This might mean that, when dealing with file/directory modules we want an inversion of the code convention: pub(world) by default, with an extension. Maybe change the prefix from pub- to local-.
• This makes a bunch of keywords somewhat redundant, of the top of my head, mod foo (it’s no implied by file structure), extern crate (now defined in cargo.toml).
  • mod {} still is useful for when you logically want a new namespace, but pragmatically want things in the same file. If anything it’s even more important with implicit files. I might want to have an implicit file contain, for example, a struct and its impls, but I also want to create some helping functions, so I’d “hide” then inside a mod type_name_internal {} to avoid contaminating the shared namespace.
  • pub use is still there, but it should become a lot rarer for crazier stuff. Hopefully it will be less common.
• I am implicitly leaving out the src/ directory, I think that tests could fit within a module (this might be more complex though). This may not be the case, but then we can easily map to something else.

Just as a final exercise, here’s how you’d do the directories for some of the examples from the article.

The future module would look something like this:

future
|- cargo.toml
|- _future
    |- pub-future // Here goes whatever stays in futures::future mod.
        |- _and_then.rs
        |- _flatten.rs
        |- ...
    |- _future.rs // This contains the Future structure, with the namespace futures::Future
    |- pub-poll_fn.rs // This is a separate module
    | ...
|- _poll.rs
|- pub-poll.rs
| ...

This allows us to have separate files with separate functionality exposed in different files. It still is somewhat complicated but it seems a bit clearer where the contents of each file is by just looking at the file structure.

Of course the question that stands out (IMHO) is: does it even make sense to keep the same structuring based for futures if this is changed? I would have to think more about this.

[dobkeratops]

r.e. these facades etc. what if there was just a way to reexport a primary item from each module, recovering a bit of the ‘class-oriented’ workflow of OOP languages; and other than that just keep the directory+file oriented module heirachy (matching the file-system).

many files are centred on one item, (vec::Vec, option::Option, hash_map::HashMap …)

if this was automatically visible one level up - and it was limited in naming to match the file (so there’s no scope for files to clash) - would that be enough?

maybe it would take some special declaration to do this, struct Self , or struct super::Baz … or it could be automatic if the names match… (struct self where self=filename might get confusing vs Self in impls where thats the current type, whereas struct super::Anything might allow clashes between siblings, not sure whats the least bad…)

imagine that , in conjunction with ‘if no mod.rs is found, just include all the files from the directory’ ; and keep the main module hierarchy matching dirs+files.

just revert to the existing system (manually defined mod.rs) to do anything fancy (per-platform conditionals etc)

qux.rs
    struct Qux{}  
    fn e(){} fn f{}
foo/
    bar.rs  
        struct Bar{} impl Bar{..}  
        fn a(){}  fn b(){}
    baz.rs  
        trait Baz{} ... 
        fn c(){}   fn d(){}

gives you these symbols:- 

    Qux
    qux::e  
    qux::f
    foo::Bar 
    foo::bar::a
    foo::bar::b 
    foo::Baz 
    foo::baz::c 
    foo::baz::d 

maybe it would make most sense to me for the ‘primary item’ and module name to be exactly the same, but that might mean bending the naming convention

[mmrath]

I have been playing with rust recently. Modules are not very intuitive definitely, I like the the proposal which is based on file structure. The current way is a bit verbose.

[ahmedcharles]

This post was originally flagged as inappropriate, perhaps because it seemed like I was attacking someone when I was really just accepting that we have fundamental differences of opinion that can’t be bridged through discourse.

One of the biggest issues I have with discussions like these is basically what @steveklabnik wrote just after this post, which is that he likes the current system but he wants to urge people to consider the struggles of others.

Sometimes things are complicated and hard to understand and usually someone will decide that they can make it better and occasionally I’m party to the pitch of how this will make everything better. I usually oppose the idea and people usually think it’s because I’m smart and I understand it and I don’t think it’s hard for me and therefore, it should be easy for everyone also. However, that’s not the case. I do think it’s hard for people. I’d never not recognize that people struggle with learning things.

However, what I do object to is people pushing through a change that will be costly and disruptive while believing that they can make it easier or better. Sure, improvements can always be made, but in a situation where you have the same constraints and all you can do is shift some things around and change some syntax, the fundamentals don’t change and they don’t get easier.

[steveklabnik]

I’m a huge proponent of Rust’s current module system.

However.

As someone who has spent the last few years talking to people who are new to Rust, I’m also forced to recognize that the module system, as it stands today, is one of the largest barriers for people new to Rust. Questions about it are some of the most common, often-asked ones. People struggle with it, a lot.

The module system makes a lot of sense to me, but it’s important to recognize that most people are not me

---

So with all that said, I am generally very pro-something when it comes to the module system. I’m not on the language team, but I’ve had a lot of talks with team members about it, iterated through many different options and designs, and generally speaking, the big parts of this proposal are something I agree with. The one thing that stuck out to me was a visceral dislike of _, but that’s sort of an immediate reaction, not a reasoned one, and also, I don’t have any better suggestions here.

---

I thought I had more to say, but I guess I don’t right now; I mostly want to urge those who are against no change here to consider the struggles of others. Some kind of reform is desperately needed here.

[DanielFath]

I thought I had more to say, but I guess I don’t right now; I mostly want to urge those who are against no change here to consider the struggles of others. Some kind of reform is desperately needed here.

Has there been any UX testing that this lessens the struggle of others?

Only thing worse than a cure, is a cure that doesn’t work.

[tomaka]

I thought I had more to say, but I guess I don’t right now; I mostly want to urge those who are against no change here to consider the struggles of others. Some kind of reform is desperately needed here.

We need some kind of figures about the proportion of people who struggle with the current system.

After all you won’t hear anything from people who understand it.

[quodlibetor]

After all you won’t hear anything from people who understand it.

The post describes issues for new users and experienced ones. I recall experiencing exactly what it describes for new users, and I currently run into the issues that it describes for experienced users.

I don’t think that the details of the proposal will pass unmodified, but the general shape seems exactly correct, to me.

FWIW, my history is mostly Python, Java, and various frontend languages.

Python has an explicit and obvious mapping to the file system. Java also has an explicit/obvious mapping. Frontend is a mess, but the import proposal has a literal filesystem mapping.

None of those categories of languages have a distinction between mod and use – I don’t recall actually experiencing that anywhere other than Rust. This proposal is more inline with Go with more hiding by default IIUC. I’m not particularly interested in getting a whole bunch of implicit imports, but there are obvious ways to make importing names explicit (e.g. use self::Struct and maybe use .::my_mod or just use self::my_mod being legal, although that would cause restrictions on the names that could appear in mods) and I’m pretty sure that if those are done then there wouldn’t be any need for modules as files, directories do everything you could want.

[rpjohnst]

Some kind of reform is desperately needed here.

Agreed.

However, I don’t believe the directory-based proposal will make it any easier- and I worry that it will make it harder, especially given the existence of the current system alongside it. _ also seems a very magical, non-obvious mechanism that’s hard to search for, as opposed to pub(crate) which is easily searchable and has analogs in other languages (e.g. internal in C#).

I’m also not convinced that the reform even needs to be primarily a change to how the module system works. Previously I suggested some new compiler messages that might make the current system easier, and others have proposed much smaller tweaks to the current system that address the concerns in the post. We could also take a look at other languages that aren’t known for confusing module systems.

For example, Python:

• A file is a module is a file, much like Rust, but without supporting mod name {}.
  • A directory must have an __init__.py file in order to be a module and contain submodules, much like Rust’s mod.rs. Other files in the directory are submodules, just like Rust.
  • __init__.py has to include an __all__ variable for from something import * to bring in any submodules. Submodules not listed can still be brought in explicitly, however.
• It has two forms for pulling in a module (import something and from something import ...).
  • import something is identical in namespacing behavior to mod something.
  • from something import * is similar to the proposed inline mod; equivalent to the current mod something; use something::*; without bringing something into scope.
• Both absolute and relative paths are used
  • in imports in they are relative to a list of search paths, much like Rust’s list of extern crates and including the main entry point.
  • in regular code they are relative to the current module

Translating this to Rust and the post’s issues, we should be able to gain a lot of familiar behavior with a much smaller (and backwards-compatible!) proposal:

• For directories containing mod.rs (or lib.rs or main.rs), automatically include neighboring files and neighboring dir/mod.rses as private submodules. This preserves single-file modules and still cuts down on declarations.
• Let mod.rs expand submodules’ visibility and assign attributes with explicit pub mod or pub(crate) mod declarations, somewhat like Python’s __all__. Error messages for this situation would look like "the module foo::something is private; declare it pub in foo/mod.rs.

For the beginner case, this brings us back from two forms (mod and use) to just one (use). Create another file, put some code in it, and either reference it directly with other_file::function or use it with use self::other_file::function or use self::other_file::*.

We lose protection from accidental extra files, which Python has because it’s dynamic and they’re never imported, but at least we don’t get accidental other directories. We also lose the ability to comment out a module declaration, but we keep something close by renaming its file extension instead (including of mod.rs).

I believe this addresses the learnability and repetition problems without changing things too drastically. extern crate could also help, and inline mod (or with this proposal maybe just use something::*) should help with the facade issue. How does this sound, to proponents of the current system and to new users?

edit: Just read the recent extern crate inference RFC. This seems to line up quite nicely with it, including the concept of adding back in extern crate/mod declarations to add visibility, aliasing, and attributes to them.