Revisiting Rust's modules

[ahmedcharles]

This post was originally flagged as inappropriate, perhaps because it seemed like I was attacking someone when I was really just accepting that we have fundamental differences of opinion that can’t be bridged through discourse.

One of the biggest issues I have with discussions like these is basically what @steveklabnik wrote just after this post, which is that he likes the current system but he wants to urge people to consider the struggles of others.

Sometimes things are complicated and hard to understand and usually someone will decide that they can make it better and occasionally I’m party to the pitch of how this will make everything better. I usually oppose the idea and people usually think it’s because I’m smart and I understand it and I don’t think it’s hard for me and therefore, it should be easy for everyone also. However, that’s not the case. I do think it’s hard for people. I’d never not recognize that people struggle with learning things.

However, what I do object to is people pushing through a change that will be costly and disruptive while believing that they can make it easier or better. Sure, improvements can always be made, but in a situation where you have the same constraints and all you can do is shift some things around and change some syntax, the fundamentals don’t change and they don’t get easier.

[steveklabnik]

I’m a huge proponent of Rust’s current module system.

However.

As someone who has spent the last few years talking to people who are new to Rust, I’m also forced to recognize that the module system, as it stands today, is one of the largest barriers for people new to Rust. Questions about it are some of the most common, often-asked ones. People struggle with it, a lot.

The module system makes a lot of sense to me, but it’s important to recognize that most people are not me

---

So with all that said, I am generally very pro-something when it comes to the module system. I’m not on the language team, but I’ve had a lot of talks with team members about it, iterated through many different options and designs, and generally speaking, the big parts of this proposal are something I agree with. The one thing that stuck out to me was a visceral dislike of _, but that’s sort of an immediate reaction, not a reasoned one, and also, I don’t have any better suggestions here.

---

I thought I had more to say, but I guess I don’t right now; I mostly want to urge those who are against no change here to consider the struggles of others. Some kind of reform is desperately needed here.

[DanielFath]

I thought I had more to say, but I guess I don’t right now; I mostly want to urge those who are against no change here to consider the struggles of others. Some kind of reform is desperately needed here.

Has there been any UX testing that this lessens the struggle of others?

Only thing worse than a cure, is a cure that doesn’t work.

[tomaka]

I thought I had more to say, but I guess I don’t right now; I mostly want to urge those who are against no change here to consider the struggles of others. Some kind of reform is desperately needed here.

We need some kind of figures about the proportion of people who struggle with the current system.

After all you won’t hear anything from people who understand it.

[quodlibetor]

After all you won’t hear anything from people who understand it.

The post describes issues for new users and experienced ones. I recall experiencing exactly what it describes for new users, and I currently run into the issues that it describes for experienced users.

I don’t think that the details of the proposal will pass unmodified, but the general shape seems exactly correct, to me.

FWIW, my history is mostly Python, Java, and various frontend languages.

Python has an explicit and obvious mapping to the file system. Java also has an explicit/obvious mapping. Frontend is a mess, but the import proposal has a literal filesystem mapping.

None of those categories of languages have a distinction between mod and use – I don’t recall actually experiencing that anywhere other than Rust. This proposal is more inline with Go with more hiding by default IIUC. I’m not particularly interested in getting a whole bunch of implicit imports, but there are obvious ways to make importing names explicit (e.g. use self::Struct and maybe use .::my_mod or just use self::my_mod being legal, although that would cause restrictions on the names that could appear in mods) and I’m pretty sure that if those are done then there wouldn’t be any need for modules as files, directories do everything you could want.

[rpjohnst]

Some kind of reform is desperately needed here.

Agreed.

However, I don’t believe the directory-based proposal will make it any easier- and I worry that it will make it harder, especially given the existence of the current system alongside it. _ also seems a very magical, non-obvious mechanism that’s hard to search for, as opposed to pub(crate) which is easily searchable and has analogs in other languages (e.g. internal in C#).

I’m also not convinced that the reform even needs to be primarily a change to how the module system works. Previously I suggested some new compiler messages that might make the current system easier, and others have proposed much smaller tweaks to the current system that address the concerns in the post. We could also take a look at other languages that aren’t known for confusing module systems.

For example, Python:

• A file is a module is a file, much like Rust, but without supporting mod name {}.
  • A directory must have an __init__.py file in order to be a module and contain submodules, much like Rust’s mod.rs. Other files in the directory are submodules, just like Rust.
  • __init__.py has to include an __all__ variable for from something import * to bring in any submodules. Submodules not listed can still be brought in explicitly, however.
• It has two forms for pulling in a module (import something and from something import ...).
  • import something is identical in namespacing behavior to mod something.
  • from something import * is similar to the proposed inline mod; equivalent to the current mod something; use something::*; without bringing something into scope.
• Both absolute and relative paths are used
  • in imports in they are relative to a list of search paths, much like Rust’s list of extern crates and including the main entry point.
  • in regular code they are relative to the current module

Translating this to Rust and the post’s issues, we should be able to gain a lot of familiar behavior with a much smaller (and backwards-compatible!) proposal:

• For directories containing mod.rs (or lib.rs or main.rs), automatically include neighboring files and neighboring dir/mod.rses as private submodules. This preserves single-file modules and still cuts down on declarations.
• Let mod.rs expand submodules’ visibility and assign attributes with explicit pub mod or pub(crate) mod declarations, somewhat like Python’s __all__. Error messages for this situation would look like "the module foo::something is private; declare it pub in foo/mod.rs.

For the beginner case, this brings us back from two forms (mod and use) to just one (use). Create another file, put some code in it, and either reference it directly with other_file::function or use it with use self::other_file::function or use self::other_file::*.

We lose protection from accidental extra files, which Python has because it’s dynamic and they’re never imported, but at least we don’t get accidental other directories. We also lose the ability to comment out a module declaration, but we keep something close by renaming its file extension instead (including of mod.rs).

I believe this addresses the learnability and repetition problems without changing things too drastically. extern crate could also help, and inline mod (or with this proposal maybe just use something::*) should help with the facade issue. How does this sound, to proponents of the current system and to new users?

edit: Just read the recent extern crate inference RFC. This seems to line up quite nicely with it, including the concept of adding back in extern crate/mod declarations to add visibility, aliasing, and attributes to them.

[tomaka]

and I currently run into the issues that it describes for experienced users.

I don’t think that the details of the proposal will pass unmodified, but the general shape seems exactly correct, to me.

What I’m pointing out is that these are all anecdotal evidences. When I say that I didn’t have trouble with the modules system, it is also anecdotal evidence. The real question is: what is the proportion of people who have troubles with the current system, and what is the proportion of people that don’t?

[Kixunil]

This is very important use case to me and I’d hate to be unable to continue using it. In some sense it’d break backwards compatibility to remove it (having stray files is allowed already).

[carols10cents]

Has there been any UX testing that this lessens the struggle of others?

Since this is a very preliminary proposal and not implemented yet, no.

We need some kind of figures about the proportion of people who struggle with the current system.

Empirical evidence is hard to get in CS; for programming language design in general, the Quorum language is the best effort I know of right now and I don’t think they’ve studied module systems.

If we were to design a UX study to compare the current system and proposed system, what would it look like? What sorts of material would we give people, and what sorts of tasks would we be asking them to complete?

There’s a lot in this thread of people with particular use cases that are complex and different from the common cases. If the common cases “proved” (as much as it’s possible to prove anything here) to be more learnable, would people accept making the complex cases more difficult? Or would only a solution that makes all cases easier be acceptable?

On a different note, I’ve seen people learning Rust who genuinely have trouble understanding the module system because the module system is confusing, and I’ve also seen people who are used to organizing their code a particular way and chafe against the common idiomatic Rust way of organizing modules. The latter group will probably resist whatever we do here unless we make whatever they’re used to the default (and since that varies from person to person, isn’t possible). I’m still digging into this proposal, but it seems like this is going even more towards “this is the convention, if you follow this, everything will work, but if you don’t, it’s going to be a struggle”. So this group might get MORE vocal, even if we make the experience better for new users who are happy to follow the conventions (once we make them more understandable). I don’t know what to do with this information exactly, except to caution that we aren’t able to get complete, empirical feedback here and complaints don’t necessarily mean we’ve made things worse.

[matthieum]

I like a lot in your proposal and much prefer it to @aturon’s.

In general, I would rather have a simple system even if it requires some boilerplate, and does not allow fine-grained encapsulation.

I’m going to build on it, by considering those related concepts in turn:

• code organization, also known as “How do I access this item?”
• privacy, also known as “Who has access to this item?”

Note: I will not address the extern crate issue; I quite like the current situation and I am not sure that it needs changing.

---

I strongly believe that code organization should, by default, be tied to the filesystem much like Python modules are:

1. Python is a widely used language, and therefore a lot of people are going to be familiar with it,
2. This de-facto eliminates a lot of boiler-plate,
3. Commenting out is easily carried out by renaming the file (changing the extension, or using a leading underscore like unused items today).

So, for code organization, I would simply:

• make the module hierarchy reflect the directory/file hierarchy,
• only include a directory if it does not start with an underscore and contains a mod.rs file (or lib.rs, bin.rs),
• only include a file if it does not start with an underscore and ends in .rs,
• error out if any “to be included” file has a name which is not a valid identifier in Rust.

This makes code organization intuitive.

This does not immediately address the Facade Pattern, and I do not think it is necessary to address it here.

---

Actually, I would even favor deprecating the mod keyword altogether.

The only other use of mod I have in my tests is #[cfg(test)] mod test and I feel like unit testing would benefit from being more keenly integrated in the language. The fact that I like small files and large test suites also means I favor putting tests in a separate file, and therefore I could easily envision:

foo/
    test/
        bar.rs
        baz.rs
    mod.rs
    bar.rs
    baz.rs
lib.rs

Then:

• foo/test/bar.rs is considered a child module of bar.rs, and therefore gets full access to its private items (access still requires super: use self::super::*; at the top is easy enough),
• If really necessary #![cfg(test)] would decorate foo/test/bar.rs, though to be honest I’d be keen on inferring it (aka, if the test directory does not contain a mod.rs, then the files inside are test files for its .rs siblings) and have the compiler report test/xxx.rs if there is no xxx.rs file to test.

Note: I much prefer this organization to Java’s, because it keeps the tests close to the code, instead of having to navigate to another subtree entirely; at the same time, it’s tidy: all tests are neatly tucked into a separate directory. Oh, and having two files means visualizing both in parallel works even in editors that do not allow opening the same file twice without contortions…

---

I am of the opinion that privacy should be the default, and public should be opt-in, for the simple reason that it forces a conscious choice of making something public, thus avoiding accidental leakage of internal details. It also does not hurt that the compiler messages can immediately spot the issue without ambiguity when trying to access a (too) private item so it’s an easy compilation error to solve.

Regarding the different levels of privacy, I am afraid that too much is too much: the vast array of choice (pub, pub(crate), pub(restricted), pub(mod), pub(self), …) is simply bewildering.

Encapsulation is certainly a desirable property, however too many options may just be that, too many. At some point there are diminishing returns eating into the language’s complexity budget. As such, I’d favor coarser granularity because it’s simpler.

I would simply use 3 levels of privacy:

• the default is private, restricted to the current module and its children,
• pub means that the item is public, immediately; that is, if foo/bar/baz.rs contains a pub struct Hello; item, then $crate::foo::bar::baz::Hello is accessible outside the crate (and each step of the path is accessible individually),
• pub(crate) is similar to pub though restricting the scope to the crate itself.

This does mean that there is no strong encapsulation within a crate; however at the same time such encapsulation would be an edit away from not existing, so I am not sure how valuable it is. I could, maybe, be convinced that pub(super) would be a useful 4th level. Maybe.

Note: I’d really have a dedicated keyword rather than pub(crate) which feels like a second-class citizen; maybe local or protected?

---

With that in mind, the facade example would require in future/mod.rs:

pub use self::{and_then::AndThen, flatten::Flatten, flatten_stream::FlattenStream, ...};

As well as each individual item being declared either pub or pub(crate) to be visible from future.rs (cannot re-export what you cannot access).

It’s a tiny bit of boilerplate, for sure, but:

• it’s literally a couple keystrokes,
• it can be made to work with glob patterns easily enough to get “inline” module (pub use sub_module::*;),
• it provides nice navigation benefits: if you have future::AndThen you go to future/mod.rs where you have a nice redirection panel pointing to either future/and_then.rs or future/and_then/mod.rs.

Note: I would favor relative paths over absolute paths, but this seems completely orthogonal.

---

Platform specific implementations require two annotations, if one wishes to paper over the differences:

• in the module itself #![cfg(unix)],
• in the containing module: #[cfg(unix)] pub use self::unix::*;

However I see the two annotations as playing distinct roles:

• the latter, decorating pub use, decides whether to re-export the symbols or not,
• the former, decorating unix.rs, allows using unix-only functions inside the module.

This does not seem completely unreasonable to me. Is it so widely common that it requires more attention?

[tomaka]

and I’ve also seen people who are used to organizing their code a particular way and chafe against the common idiomatic Rust way of organizing modules

Do you have an example?

The only thing that comes to my mind is these sort of weird reexports, which are clearly unidiomatic. But I’ve only ever seen this in the stdlib (and I guess it’s being kept because it would be too annoying to fix) and never in any third-party library.

[U_007D]

Wonderful discussion, everyone. Thank you to the Rust Language team for sharing your thoughts and for requesting ours.

In terms of the proposal, I’ve found that Rust’s “convention over configuration” policy to be pretty compelling, as long as the convention is well-documented and easily discoverable. The proposal put forward by @aturon and @withoutboats, at a high level, seems to be another good example of this–I like it.

As a newb to Rust and as one who has (and still does) struggle with modules, there are a couple small things that would go a long way to clarifying the module system for me.

1. Outside of Rust, module is often used to mean “generic piece of code”, similarly to the way node refers to a generic data element in a structure. This means that mod does not serve as a good mnemonic (at least for me) for what is happening when I use it–in fact, after “getting it”, I came back to writing in Rust after a short two-week vacation, and could not recall whether I needed to mod or use a module to “import” it. Aliasing it to import would certainly help me to understand/remember what the mod keyword does.

2. I recently ran into a case where my lib.rs sits alongside my main.rs in src and I’ve created an args.rs module, an error.rs module and a consts.rs module all referenced by lib.rs. I decided I wanted to hide all this ‘noise’ in the root directory and tried to move lib.rs to lib/mod.rs. I’d hoped to put args.rs, error.rs and consts.rs into lib, alongside mod.rs. But this didn’t work. Apparently as a crate name, lib.rs is hard-coded. I feel strongly that orthogonality of rules is very important, so we don’t have to say that a system behaves in such-and-such way except for when…

So what does integrating the proposal into the issues I’ve run into look like?

1. mod's lack of mnemonicness goes away as a problem. There’s no import (keyword with an explicit mnemonic), but that’s probably OK–no new keyword, no redundant keyword, easier to teach, etc. From everything I’ve had to do so far, my code would have less header boilerplate, so that’s a win.

2. The proposal does not address the hardcoded crate expectations. I would like to see the special ‘hardcoded’ src/lib.rs (and presumably src/main.rs for binaries) loosened to lib.rs or lib/mod.rs (+ equivalent for binary crates) – the same rules that apply everywhere else.

With the above, at the highest level, the proposal seems handle all the use cases I’ve run into in my short time with Rust, while reducing the code needed to build modular systems. And that’s pretty attractive from where I stand.

Thanks again, @aturon, -Brad

[aturon]

Wonderful discussion, everyone. Thank you to the Rust Language team for sharing your thoughts and for requesting ours.

Just a quick aside – I really appreciate you saying this, and would’ve even if you’d come out against the proposal. It can be pretty demoralizing reading numerous comments that begin with “I disagree with everything you wrote”

[rpjohnst]

I would simply use 3 levels of privacy:

I like this, I haven’t personally run into a need for pub(super) or pub(in foo). But I’m not opposed to what’s currently specified, since the pub() notation makes it pretty clear that it’s just a variation on pub visibility. I haven’t followed the pub(restricted) RFC enough to know what use cases people had in mind.

(For reference C# uses internal as its equivalent to pub(crate))

[le-jzr]

@aturon About “Path confusion”: To me the greatest confusion is not the inconsistency between use and everything else, but the inconsistency between different cases of use. use std::*; is relative to the root, while use self::*; and use super::*; is relative to the current crate. This one was confusing me big time, and sometimes still is. Though I guess that’s just a symptom of it being the opposite of the traditional arrangement.

I love the rest of the post. It would be amazing if modules worked like that.

[liigo]

use std::*; is relative to the root, while use self::*; and use super::*; is relative to the current crate

std is imported/bound to the root of the current crate

[ihrwein]

@aturon: I really like the core of the proposal and the intention behind it - make Rust easier to learn and use. I’ve just one nitpick: pub(mod) seems to be more reasonable than pub(self) (I don’t like to use self outside of methods). This proposal combined with the “deduced extern crate” RFC could eliminate a ton of noisy use declarations.

[est31]

It seems impossible to maximize ergonomics for both patterns.

I disagree. See the inline mod proposal which is IMO the most favorable solution, and works without having to use these ugly _ characters.

[DanielFath]

If we were to design a UX study to compare the current system and proposed system, what would it look like? What sorts of material would we give people, and what sorts of tasks would we be asking them to complete?

I’m no UX experts. But here is how I understand the problem. We need to first get answers for some questions:

1. What are the tasks that are most confusing for users?
2. What type of users have these problems (novice/intermediate/experienced)?
3. What do they most likely complain about?
4. What is their mental model of module system?

Answering #4 will tell us how to make a new model. Then using those questions we could set up some kind of tests. Since it seems our audience is newcommers, we could possibly try doing some A/B tests when teaching, looking for which group has more trouble learning.

However this only gets us half-way and we should see if experienced users are impacted.

---

My personal thoughts on aturon’s idea is that it seems good, and is very Pythonic, but I’m not sure if it will sit well with people not familiar with Python.

[le-jzr]

My personal thoughts on aturon’s idea is that it seems good, and is very Pythonic, but I’m not sure if it will sit well with people not familiar with Python.

I’m not familiar with Python, but I find the system very natural. Much more so than the current one, which forces you to deal with modules and reexports whenever you want to just split code into multiple files. Organizing code using multiple files is ubiquitous, so this is a pretty major papercut.

Incidentally, the proposed system is similar to what Go is doing as well.