[lang-team-minutes] the module system and inverting the meaning of public

[tshepang]

That said, I wish core team should start planning for 2.0. We should not get stuck with stuff we don’t want, just because we want certain people to take us seriously. With continued use, we will discover lots of things that are not ideal. How many collective hours do people spend dealing with not-ideal parts of the language, and working around them in order to preserve backcompat? We also got tools that can detect if things are deprecated and can provide suggestions, and that’s a hint at automated conversions. This should make things a lot easier than Python 3 (as an example).

I wish Rust does not become a victim of its own success… “But, but what about all these users? For one, there’s so much Rust code in Firefox. Think of Dropbox as well. If we break, they won’t take us seriuosly.”

[zackw]

As a bit of general perspective, Rust is what, seven years old, counting from Graydon’s announcement at the Mozilla all-hands in Whistler, which I think was 2010? The 1989 C standard was nearly thirty years ago, the language itself is nearly fifty years old, and C89 was a huge breaking change which some people are still carrying around compatibility #ifdefs for.

I do think C89 levels of stability are a good long-term goal, but by that metric we have another 13 years to get there.

(I haven’t been paying any attention to what’s going on with C++ these days so someone else can make that comparison.)

[ahmedcharles]

Today, when you declare a struct, you also give it some associated privacy. Using the pub(restricted) rules, structs can either be declared as “world accessible” (pub) or “private to some module in your crate”. The default is the to private to the current module, but one can write pub(super) to be “private to the parent module”, pub(crate) to be “private to the root of the crate”, or pub(in foo::bar) to be “private to the module foo::bar”. Being “private to a module” means, roughly, that nobody outside of that module can name the type or possess an instance of that type, except in generic code. The “private type in public API” rules are intended to enforce that guarantee (but they have issues of their own that we are still working through). We named this interpretation the <= interpretation, because when you declare a struct as being public to a module M, you are saying that it is accessible only to submodules of M (i.e., modules “less than” M).

This seems internally inconsistent, in that it defines the various types of visibility as private type in module X and then switches to public to a module M. Being public to module M isn’t defined, that I can tell? That and various other minor inconsistencies make it hard for me to understand the proposal completely.

At a high level, I have a hard time understanding why people have a difficult time with the module system and whether it’s the system that’s too complex. A position I take that probably makes me sound elitist is that some things are complex and you just have to try harder to understand them. For example, I first read the recursive page table mapping in http://os.phil-opp.com/modifying-page-tables.html when it was first written over a year ago and I didn’t understand it then. I eventually reread it and understood it a few weeks ago. Complaining that paging is too hard wouldn’t make it less hard at a fundamental level. Anyways, that’s just anecdotal.

In the current case, explicitly declaring a module using mod foo seems to represent a fundamental aspect of the module system. In terms of system programming languages with module systems (at least proposed ones), I only know of Rust and C++. In both cases, you have to declare them, explicitly and I think the Rust system is easier to understand. I also happen to prefer explicit over implicit.

because use statements are tracked from the crate root. (I would love to figure out how to solve this but haven’t found a backwards compatible solution that isn’t even more confusing).

At a high level, every language that wants to see wide adoption has to stop changing things at some point and after that point, bugs become features. I would agree that having names in use statements be based on the crate root while other names be based on the current scope is weird and confusing, but it’s no where near as bad as other languages. And trying to fix it can also end badly, in my opinion. Anecdotally, C++ had the ‘most vexing parse’ which uniform initialization was meant to fix. But it amusingly created another parsing ambiguity that’s even more obscure with enums while making it backwards incompatible to add an initializer_list constructor for a class after it’s already being used, in the general case.

Sometimes when you try to fix things, you just make them worse. Hopefully we can avoid that.

[kornel]

No, use of modules isn’t an inherently complex problem. Rust just has overlapping functionality of create, mod and use, a mix of absolute and relative paths using confusingly similar syntax (which also happens to have a special case in crate root), which all complicates things for reasons that are minor and/or arbitrary, and not inherent to modularity.

In CommonJS there’s a single require that combines role of both mod and use, it is explicit, and it is easy to understand.

[ahmedcharles]

I never said that modules were inherently complex. I said that declaring a module to exist is a fundamental aspect of the module system. I personally think that the module system as it exists today is fine, ignoring the <=, >= and == stuff that is not stabilized yet. The use keyword, which is technically not required (reexporting can be done in other ways for types and functions, though not easily for modules), has weird semantics, because it does name lookup differently than everything else in the same file (except for when writing the crate root). The mod statement however, is simple and does what someone would expect after reading the ‘book’. And the extern crate statement is even more niche, but also simple and required.

It probably seems like I’m being dismissive of people who have a hard time learning the module system. However, my point is that there is a point at which things become difficult for everyone to understand. If that’s the module system in Rust for some people, that doesn’t suggest that the modules system needs to be changed. And changing it to make things implicit seems like the wrong answer. Almost everything that’s hard for me to learn and reason about in languages I’ve used are due to the implicit features of the language. Perhaps having more explicit things increases the initial investment and learning curve for things that need to be typed (or perhaps you just copy them from somewhere else until it compiles), but ideally, this additional knowledge helps with learning the language long term.

[kornel]

It probably seems like I’m being dismissive of people who have a hard time learning the module system. However, my point is that there is a point at which things become difficult for everyone to understand. If that’s the module system in Rust for some people, that doesn’t suggest that the modules system needs to be changed.

The #1 goal for Rust (edit: this year) is to lower the learning curve. If people hit modules on their learning curve, then that is the reason to make modules simpler to use.

While mod foo {} in itself sounds easy, it doesn’t exist in a vacuum. It exists alongside use, and they are not orthogonal features, which makes it confusing, e.g.

• in the root of the crate you write mod foo;, and can’t write use foo;,
• but elsewhere you write use foo;, and can’t write mod foo;.

This is specific to Rust, and it looks weird and arbitrary from perspective of JS, Python or Java, which don’t have this distinction.

I stumbled upon this, and it just added to my frustration with Rust. I did read the book, I did ask on IRC. Eventually I got it, but it was a waste of time for everybody.

Rust can’t exist in practice without use, but mod can easily be removed from the list of things a novice has to know in order to use the language.

[Storyyeller]

One thing I’ve been wondering is whether explicitly telling the compiler which source files to include is really worth all the misery that it causes. Go just includes all source files in a given directory, and it seems to do just fine.

[yigal100]

The pub(module) distinction this relies on is very confusing and uses ugly syntax.

I do understand the sentiment that having more refined horizons inside the crate might help refactoring but I get this gut feeling that this is the wrong tool for the job. This is better served by other means such as IDEs, using smaller crates, making concise and small modules and files, etc…

This smells to me exactly like friend classes in C++ which is better avoided in practice.

I think most popular languages stick with a mostly three-part system and I think we could adapt a similar approach:

1. private - implementation details inside this module 2a. something akin to java’s default “package” which is accessible by parent module and siblings 2b. something akin to C# internal which is assembly scoped or Crate scoped in Rust land.
2. public - world accessible.

I’m not sure which of the #2 options is better or maybe we need both. perhaps, we can say that “public” is crate local and the user can “export” APIs with pub use

[est31]

I’m a great fan of pub(restricted) where you can make things public on a crate level. I disagree though with the proposal to make pub sth inside a module public to the world. This would be a breaking change. How long was Rust 1.0 ago? Not much. I don’t want to rewrite my program, especially not for such a minor change that isn’t even very useful (in fact I don’t like it at all).

Also I don’t like the argument “its confusing for beginners”, as for me it wasn’t confusing.

In fact, it would be now far more confusing, as the proposed system would introduce a big inconsistency:

mod foo {
    pub struct Thing {} // This is world readable
}

fn foo() {
    pub struct Thing {} // This is not world readable!
}

[ahmedcharles]

The rust-lang.org page says: Rust is a systems programming language that runs blazingly fast, prevents segfaults, and guarantees thread safety.

That is basically: safety, performance, concurrency; pick three.

That’s the number one goal of rust, though at this point, it’s just splitting hairs. The #1 focus of Rust development in 2017 may be improving the learning curve, but that goal isn’t at the expense of everything else. If you want a language that’s easy to learn and sacrifices any of the three things above, I’d recommend Lua.

At a high level:

1. I don’t personally think the Rust learning curve is that bad and the places I think it happens to be bad is in the places where people made changes to the language to make the learning curve better, like autoderef, as an example.

2. Even if I did buy into the idea that the learning curve is bad, I wouldn’t focus on changes to the language or the module system. That would primarily result in instability and confusion. I’d focus on documentation and other educational initiatives. And where I did make language changes in this vain, I’d have to prove to myself that making the change will improve the experience for the vast majority of Rust users, at all skill levels.

3. Even if I did believe that changing the language was appropriate and modules was a big problem, I don’t see how either of these changes makes it easier for Rust users at all skill levels to learn Rust in the long term.

About mod specifically:

You can write mod foo; in a non-root crate, it just refers to src/**/foo/mod.rs or equivalent instead of src/foo.rs or equivalent. You can write use <identifier>; in the root crate at all, because any individual identifier you can put there is already imported. But that issue is one of name lookup and has nothing to do with mod. mod doesn’t do name lookup, it introduces a name.

Rust has 2 core concepts and 1 auxiliary concept:

1. Declaring names in the current scope: mod foo, fn foo, struct foo, enum foo, etc.
2. Using names from the current scope: foo::bar, foo(), foo {}, foo::variant, etc.
3. Using names from the current scope and declaring them as different names in the current scope: use foo::bar, extern crate foo as bar

As far as I can tell, use statements are for convenience. Perhaps they are required in corner cases, but if you’re willing to type the entire name every time, you don’t need them. Reexporting names using pub use isn’t a beginner feature anyways.

If someone doesn’t understand the distinction between the three scenarios above, then it doesn’t matter what the syntax is, they’ll have issues with Rust. And hiding mod from them isn’t likely to help or even delay the learning curve in a useful way.

And as for other languages:

JavaScript seems to require module.exports to declare a module. Python seems to require a __init__.py file in a directory and Java requires package statements. They are all different ways if indicating that the module system is in use and they are different from the way Rust does the same. I don’t see how they impact this discussion at all, other than to suggest that there are huge inconsistencies and misconceptions between languages and their module systems.

As for whether a novice needs to know mod, it depends on when you want to teach them code organization. When would someone in JS learn about modules.exports? Or a Python programmer learn about __init__.py? Or package in Java? None of those things are part of tutorials because code organizations isn’t usually part of a language tutorial. I don’t need mod in Rust to write a single file program as part of a tutorial, so why is it part of the critical path?

[ahmedcharles]

And just to be clearer about comparisons to other languages, every language I know that has a module system, requires something explicit at the declaration of the module which would not otherwise be required by the language to be valid code. Sometimes this is additional text in the file declaring the code for the module (JS, Java and Lua). Sometimes it’s an extra file (Python). Sometimes it’s additional text in the file which represents the ‘parent’ of the module. But in all cases, you can’t take a file with normal code and turn it into a module without doing something explicit.

[kornel]

Changing an error unresolved import 'foo' to find the module foo instead does not compromise Rust’s safety, performance or concurrency. It does not break backwards compatibility. But it does help users avoid an error and progress with use of Rust further with one concept less to learn.

use statements are for convenience

No, they’re essential for traits. Rust relies on them for basic things like file I/O. Rust without use statements is impractical, so teaching of use is unfortunately unavoidable.

However, Rust could be made usable, and taught to beginners, without mod. If use implied mod, then novice users would not need to learn about mod to progress from single-file programs to multi-file programs.

Rust has 2 core concepts and 1 auxiliary concept:

The problem is that other languages have merged/simplified these concepts for modules. You can do const foo = require('./foo') without declaring existence of foo first.

You mention syntax like module.exports and package. These declarations are different from Rust’s way of thinking, because they are in the module file. This is another way Rust’s module system can be misunderstood. One could put mod foo in foo.rs mistakenly thinking that mod foo marks the file containing it as the module called foo.

mod.rs is analoguous to __init__.py and index.js, so that aspect of modules is actually similar to what other languages do. I do not suggest changing that part.

I don’t personally think the Rust learning curve is that bad

Saying it’s not bad for you does not help users for whom it is bad. But making a concept unnecessary does help users who don’t understand/misunderstand it.

[ahmedcharles]

Changing an error unresolved import ‘foo’ to find the module foo instead does not compromise Rust’s safety, performance or concurrency. It does not break backwards compatibility. But it does help users avoid an error and progress with use of Rust further with one concept less to learn.

That’s what’s called a quality of implementation issue. There could be a warning that notices a foo.rs next to lib.rs and indicates that lib.rs should have mod foo; in it somewhere. And similarly, if lib.rs contains use foo; while there exists a foo.rs with no mod foo; in lib.rs, tell the user that use foo; should be mod foo;. If those warnings don’t exist and they aren’t in the process of being implemented, then I consider complaining about the usability of module system to be almost a self fulfilling prophecy. Obviously if you don’t have the best possible warnings and errors guiding the user, it’s going to be hard to use/learn. I don’t see why we would change the language before addressing the quality of implementation of the current language spec and see if that improves usability enough to move on with other things.

No, they’re essential for traits. Rust relies on them for basic things like file I/O. Rust without use statements is impractical, so teaching of use is unfortunately unavoidable.

UFCS doesn’t require use statements. I’m not suggesting that UFCS should be taught to beginners, but my statement is accurate. It’s also the case that the most common traits being in the prelude means that at least the initial programs being taught and explained don’t require use or UFCS at all.

However, Rust could be made usable, and taught to beginners, without mod. If use implied mod, then novice users would not need to learn about mod to progress from single-file programs to multi-file programs.

If use is taught before multi-file programming is introduced and mod is avoided before that, then multi-file programming can be introduced along with mod while introducing no other new concepts. That seems like the ideal scenario for adding mod to a user’s knowledge base with the greatest chance of them not being confused.

The problem is that other languages have merged/simplified these concepts for modules. You can do const foo = require(’./foo’) without declaring existence of foo first.

While I’m not a JS programmer, my understanding is that in order to do const foo = require('./foo') someone else is required to have typed modules.export = .... The appropriate analogy of that in Rust would be extern crate foo;, which would result in the user being able to use foo, just like in JavaScript. You seem to be trying to compare a JS user who uses another person’s module with a Rust user writing their own module. Obviously, those are significantly different user experiences and it is not fair to compare them.

You mention syntax like module.exports and package. These declarations are different from Rust’s way of thinking, because they are in the module file. This is another way Rust’s module system can be misunderstood. One could put mod foo in foo.rs mistakenly thinking that mod foo marks the file containing it as the module called foo.

You are correct, it is different than other languages. But then again, Java, Python, Lua and JavaScript all have different models for modules and knowing any given module system prepares one to learn any of the others. Rust isn’t special here, so I don’t see why comparing it to another language is interesting after stabilization. The best case we could achieve is copying a system from an existing language and making Rust as close as possible, but that would only benefit the learning curve of users with a single language as a background, which is obviously not an interesting goal at this point in the language’s evolution.

Saying it’s not bad for you does not help users for whom it is bad. But making a concept unnecessary does help users who don’t understand/misunderstand it.

As a Rust user, I get a ‘vote’ and an ‘opinion’ in whatever process is used for community governance, much like you get an opinion and a vote. If people are allowed to say that they find the system complex and hard to learn, I’m allowed to indicate that my experience was the opposite. That’s all.

[Storyyeller]

But then again, Java, Python, Lua and JavaScript all have different models for modules and knowing any given module system prepares one to learn any of the others. Rust isn’t special here

Rust is special in the sense that its module system is unlike any other language, and not in a good way.

Compared to Python, Java, Go, etc. Rust is the only language that requires separate statements to declare the existence of a file and to import it. The closest analogue I can think of is C++, but in C++ #import and namespaces are completely orthogonal, and I don’t think anyone would praise C++'s import system anyway.

Anyway, I’m not sure to what extent it can be changed backwards compatibly, but removing the requirement for redundant mod statements should be backwards compatible and is an obvious ergonomics and beginner friendlyness win,

[ahmedcharles]

@Storyyeller

Compared to Python, Java, Go, etc. Rust is the only language that requires separate statements to declare the existence of a file and to import it. The closest analogue I can think of is C++, but in C++ #import and namespaces are completely orthogonal, and I don't think anyone would praise C++'s import system anyway.

Overall, I'm not sure why comparing languages is relevant to the discussion, when all of them require an explicit indication when a module is created. And more importantly, all of them are different.

@Storyyeller

Anyway, I'm not sure to what extent it can be changed backwards compatibly, but removing the requirement for redundant mod statements should be backwards compatible and is an obvious ergonomics and beginner friendlyness win,

I don't think it's an obvious ergonomics and beginner friendliness win.

I agree that Rust is different. But that's not interesting.

Though, it seems like we're talking past each other.

My primary point is: The current implementation of the spec lacks quality of implementation features which do not require changing the language to implement and try to improve. Why is the initial suggestion, a change to the language? In most mature languages (which Rust wants to become soon), changes to the language are rare and only done when all other avenues are exhausted. I'm never going to suggest this proposal should never be added to the language, but I fail to understand why changing the language is the first thing proposed. Note: This seems to be common here, because almost every thread is suggesting a language feature or change to fix some papercut, when other alternatives are not considered or discussed. Perhaps because people view this as a language design discussion board rather than a place to discuss the evolution of a language and it's ecosystem.

Anyways, does it require an RFC to add a warning to rustc? If not, would anyone like to mentor me adding warnings to rustc which would help teach users about the differences between use and mod when they get it wrong? If users still complain about the distinction after we've run out of appropriate warnings to add and documentation to adjust, perhaps a change is warranted, but I'd much rather avoid adding more implicit things to the language that people get to avoid and not learn just so we can check a usability box on a 2017 roadmap post.

[ahmedcharles]

I created https://github.com/rust-lang/book/issues/460 which is an issue against the new version of the book to address what I think is confusing wording which may lead people to misunderstand the distinction between use and mod.

[steveklabnik]

Anyways, does it require an RFC to add a warning to rustc?

Yes, in general, new lints require an RFC. This is for a number of reasons, but one of them is that they’re likely to break existing code.

clippy, however, adds new lints all the time, and does not.

[withoutboats]

I haven’t followed the discussion super closely but I think @ahmedcharles is actually talking about adding more notes to existing error messages, which doesn’t require an RFC.

[ahmedcharles]

Well, compared to changing the semantics of the language, lints seem like less heavy-weight approach and adding lints to clippy wouldn’t be a solution to the problem that this proposal attempts to solve, so RFC it is?

[ahmedcharles]

I think there would be at least one new warning, which would look for files which do not have mod statements and suggest adding them.

The rest can probably be done by improving existing warnings/errors.