[Pre-RFC] Yet another take on modules

[newpavlov]

I would like to present an evolution of ideas presented in the comment to the second module proposal. This proposal tries to incorporate ideas from other posts (and respective comments) and to find a reasonable compromise on some issues. It still has some holes in the reasoning and description, but I hope it will help with the further discussion of the topic.

Other proposals and discussions

1. Revisiting Rust’s modules
2. Revisiting Rust’s modules, part 2
3. Revisiting modules, take 3
4. pre-RFC: inline mod
5. pre-RFC: from crate use item

Abstract model

We can model module system as a way to construct a directed graph (possibly with cycles, see open questions) with two types of nodes: item and module. “Item” type covers not only usual visible items: structs, traits, functions and others, but also “invisible” as well, e.g. impl blocks or trait implementations, which should be linked to the graph to be usable. “Module” type essentially just a way to organize access paths to the items from the given “module” node. (e.g. from the root represented by lib.rs, which will be named as “crate” from here) Note that one item can be accessed through different paths. (e.g. crate::foo::bar::zoo::Item and crate::prelude::Item)

Edges have only one type. All nodes and edges should be organised in a such way that all “item” nodes should be reachable from the “crate” node.

Nodes and edges can be marked as “exported”. Root node is always marked as “exported”. All “exported” nodes and edges should form a connected graph (note: not tree), which will be called “exported” graph.

We can link external “exported” graphs from our graph by creating edges to its nodes.

In the code

To represent the graph described earlier we will need the following tools:

• use path::to::node; – main tool for creation of edges (linking). Path is relative to the module in which it’s used. If needed will attempt to create nodes and edges by following files and directories on the file system. Absolute paths represented using reserved node name crate which represents “root” node. (usually lib.rs, but can be overwritten by Cargo.toml) So absolute import will look like use crate::foo::bar;.
• from cratename import path::to::node; – create an edge to an external “exported” graph from crate listed in the Cargo.toml dependencies section with absolute path inside of it. path::to::node should go over nodes and edges marked as “exported”.
• export path::to::node; – link node to the current module and mark it as “exported”. Node can be marked as exported more than once. path and to stay unchanged. (so if they were not marked as “exported” they stay unexported) Edge current_module -> node marked as “exported”.

While it’s enough to have tools described earlier it’s certainly inconvenient to use just them. Thus we need additional tools usable inside mod.rs and lib.rs:

• use(auto) [self]; – will desugar into use {foo, bar, ...}; where foo.rs, bar.rs and others are files in the same directory.
• use(auto) foo; – will link modules defined in the files inside foo subdirectory, which should not contain mod.rs.
• use(inline) [self]; – will link all public items inside modules defined by the files in the same directory, but will not link those modules. So “module” node defined by the foo.rs will not be reachable from the root, but all public items inside it will be linked to the current “module” node.
• use(inline) foo; – “inline” files inside foo subdirectory, which should not contain mod.rs.
• export(auto/inline) [self, foo]; – behaves exactly like uses, but additionally marks linked nodes and respective edges as “exported”.
• import crate_name; – link root node of external crate. (analogue of import crate_name import self;)

Dangling “exported” node (i.e. node without path from crate node passing through nodes only marked as “exported”) will trigger a warning while compiling, as it will not be accessible for user of the crate.

We can “export” nodes (items and modules) imported from another crate too, so this is a correct code: import foo;

Privacy

Only items marked as pub can be marked as exported, otherwise it will result in the compilation error. pub implies pub(crate). Stricter privacy constrains can be placed on items which will limit edges creation and referring through paths.

Common patterns

Some common library patterns and how they will look under this proposal. (section will be updated based on discussion)

Facade

export(inline); fully covers this use-case and additionally allows more flexible directory structure. E.g. using features example from the first @aturon post we can write:

export(inline) {self, flatten, map, select};

Here flatten, map and select are subdirectories which contain flatten.rs, flatten_stream.rs, map.rs, etc. lib.rs will contain export future;

Prelude

We have crate with the following items (all path points marked as “exported”): foo::t:Item1, foo::t2::Item2, bar::Item3. For convenience we want to expose them through prelude, so users could import them as from my_crate import prelude::*;. To implement it in lib.rs we need to use export prelude; and in prelude/mod.rs:

export crate::foo::t1::Item1;
export crate::foo::t2::Item2;
export crate::bar::Item3;

Or if we really want to abuse multi-line:

export crate::{
   bar::Item3,
   foo::{
      t1::Item1,
      t2::Item2,
   }
}

Re-factoring single file into folder

For example we have foo.rs containing Item1 and Item2 which became too large. We want to create a folder foo with the code for those items. We can just move foo.rs to foo/foo.rs, create file foo/bar.rs and copy Item2 code to it. Now inside lib.rs if we had export foo::{Item1, Item2}, we can change it to export(inline) foo, or to export foo::{foo::Item1, bar::item2}.

Simple crate

lib.rs:

export foo;

foo.rs:

export Item;

pub struct Item;

bar/mod.rs:

export(auto);

bar/zoo.rs

// Will not be accessible as crate::bar::zoo::Item in the exported graph
use crate::foo::Item;

pub struct Zoo;

Item will be accessible through from simple_crate import foo::Item; and Zoo through from simple_crate import bar::zoo::Item;

Pros and cons

Pros

• Clear design with relatively small number of rules
• Explicit implicitness of using folders and files for defining modules.
• Flexibility, allows fully explicit or highly implicit approaches
• Intuitively covers common use-cases
• Easier to teach
• Solves “automatic promotion of key items” problem
• No conflict between simultaneous lib.rs and main.rs

Cons

• New keywords
• Significant breakage compared to the current system
• auto and inline will certainly be most common, which will encourage implicitness across ecosystem
• In depth explanation can be a bit tricky due to the more complex model

QA

Why “from foo import bar” and not “from foo use bar”?

We already breaking use pattern with the introduction of from keyword, so the only reason to keep use is for backward compatibility. Meanwhile introduction of export keyword makes import feel quite natural, while also allowing convenient shortcuts like import cratename; instead of from cratename import/use self;. And of course it will feel very familiar to people coming from Python, which one of the main sources of Rust grow.

Don’t we place too much functionality on use keyword?

Probably yes, in this proposal use and by extent export can be use for two things: linking source files from file system and creation of “shortcut” links. Initially I thought about using mod (plus mod(auto) and mod(inline)) for linking files and convenience combination export mod(auto/inline) [foo] for exporting stuff, while using use keyword only for “shortcut” edges creation. I like such design a bit more as we explicitly define two types of edges on module graph, with mod edges required to form a tree, it simplifies some things conceptually, but makes it a bit harder to learn. (beginners confusion between mod and use which we witness today) But looking at a general direction of other proposals, I’ve decided to remove mod as well. Although this decision can change based on discussion.

Open questions

• How to correctly define super nodes in the given model.
• Rules regarding “bad practices” of using auto and inline.
• Can we do without restriction on mod.rs for directories linked through use(auto/inline) foo?
• Change keyword usage to be more backward compatible?
• crate::foo::bar vs ::foo::bar
• Should we keep mod keyword for example for inline modules?
• Should we explicitly forbid creation of cycles in the module graph or should we allow it?

[matthieum]

I’ll be honest; I prefer explicitly listing the items being exposed (and where they come from).

I fill that by facilitating the Facade pattern too much, the link between the point of use and the point of definition is lost, making it more difficult for the human to trace it.

I know that IDE feature go-to-definition, and it’s a fine short-cut. I would however prefer if it was not required to have an IDE, and instead it were possible to simply follow Ariadne’s thread.

Note: this is also the reason why I am not such a fan of glob imports.

[newpavlov]

I agree with you, this is why I wrote that we’ll need rules for determining when inline and auto can be used and when probably they should not be. But as I see it there is a significant push to introduce such implicitness to the language, so it’s quite probable that in some form or another it will find way to the next version of module system.