Revisiting Rust's modules

[arielb1]

but it still felt like the approach was “pointer syntax must be removed from patterns” rather than the actual cited motivation of “the confusion between ref and & in patterns must be cleared up.”

The motivation was quite openly “playing type tetris in patterns is annoying”. The “type tetris might confuse beginners” was secondary.

[rpjohnst]

…sigh. The approach still felt like “pointer syntax must be removed” rather than “type tetris must be made less annoying.”

[ahmedcharles]

One specific question before I give my higher level thoughts on things: What do you think about the inline mod proposal that I made?

One of the things that I think can explain why there is frustration or a feeling of us vs them, boils down to opportunity cost. Having members of the official teams spend time on something is a huge opportunity cost and additionally, any proposal made by a member of one of the official teams is likely enough to get accepted, that it’s an opportunity cost forced on anyone who disagrees.

So, if a member of one of the official teams makes a proposal to solve a problem which people can acknowledge but they can justify as not being sufficiently important or urgent to be addressed by that person, that can make it seem like the system is biased or that the decision on the issue is a foregone conclusion.

To take the module system as an example, many people seem to have conflicting views and occasionally, the same person. For example, some believe that the module system is more complicated than it’s importance to the language. Well, without the module system, the language would only have a single global scope, like C, ignoring lexical scopes created by blocks. And name lookup is a part of any language that I think has hidden complexity that is often unappreciated, so it’s not surprising that people find Rust’s module system to be complicated. It’s a fundamentally complex thing that Rust treats differently than most languages. But ironically, if one viewed the module system as not very important, why would it be discussed so heavily?

In conclusion, part of the reason I think people feel like they have no voice and aren’t being heard is that the discussion is still taking place and it’s not going anywhere and rather than switch priorities to something less contentious, it just continues to waste time.

[edmccard]

why would file-level be interesting at all?

For one reason, because we already have it – when a file contains a single module, anything private to that module is also private to that file. For another reason, because (until a new way of writing code becomes widespread) files are natural boundaries – the distinction between “code I’m working on” and “code I’m ignoring for now” maps onto the distinction between “files I have open” and “files I don’t”.

[withoutboats]

What do you think about the inline mod proposal that I made?

We’ve considered a semantically equivalent idea before called ‘anonymous modules’ (the exact syntax was different). In general I personally like this idea; it evolved into the idea in the blog post when we reviewed and found that the majority of modules in many crates would be inline/anonymous.

[dpc]

For example, I fully acknowledge that a downside of reading module structure from the file system is that stray file cause problems, which in turn invalidates some existing workflows.

It is actually not about workflows. It is about automation and practical reliability. My workflows are going to be OK. The compiler will tell me stuff doesn’t compile, I will see a leftover file, move it or whatever. That’s not a problem at all.

The problem is that typically code lands in one big mono-repo, between plenty of other programming languages, all glued together by in-house top-level building system etc. There are files being generated, sharing API definitions, schemas, protocols, what not. Then people rebase their branches and suddenly language X code fails to compile for reasons unknown, and the person that got the failure is Y programming language developer, and has no idea what happened. And that person gets an immediate reaction: “hmmm… that language X is a flaky language failing like that, another reason not to use it”. And that person asks on Slack, and the team responds: “oh, it’s just probably stale file, do a clean build”. And while it is the building system being at fault here, for not handling auto-generated files correctly, etc., in practice it is a hard problem to prevent such cases, it’s hard to fix, and building systems that don’t suck are very rare. Meanwhile, C/C++ never behave like that, no matter how archaic and backward the module system they have.

The above example is not made up. That person is me, and language X is Scala, which is otherwise quite a fine language. And language Y is Rust, and I think to myself … I wish Scala had mods like Rust does…

Flaky problems are 100 times worse than consistent problems. I would rather deal with a language that forces me to do something pointless every time, and learn to live with it (Stockholm Syndrom is such a bliss…), than a language that once in a hundred times makes me figure out something pointless. So in context of module system, not implicitness is the danger here. By offloading module information to file system, we introduce another dependency (file system content) that is involved in the result of compilation. I could even imagine security attacks that rely on dropping files into source repository etc.

[Ixrec]

I’m also very concerned by the “sense of polarization” aturon’s referring to, so here’s my quasi-outsider perspective on how the “ergonomics RFCs” this year have gone.

Overall, I agree there is a sense of inevitability about some of them, but in a good way. I do not think the lang team has been ignoring the “don’t do this, it’s not worth it” perspective. Rather, I think these RFCs felt inevitable because the lang team did a good job choosing the “low-hanging fruit” in terms of possible ergonomics improvements. These were mostly issues where, from at least the start of this year, there already was a growing sentiment if not a sort of consensus in the community that the problem was real or the syntax was unhelpfully redundant or the non-sugary solution was tedious or whatever. And many of them were “papercuts”, i.e. the actual problem is relatively small in scope so all possible solutions to the problem are going to look relatively similar, at least superficially, not because alternatives aren’t being considered but because the space of possible solutions simply isn’t that large.

• The match ergonomics RFCs. The two RFCs are extremely different in the implementation, and as far as I know, nobody ever suggested any alternative solutions to the problem that did not involve removing the need for typing ref in patterns. This thread is actually the very first time I’ve even heard it suggested that there might be a way to make ref more ergonomic without simply removing it; all the high-level objections I recall from the RFC threads were of the “don’t solve it at all, it’s just not worth it” variety. Plus, I honestly believed this RFC was dead in the water for a while after the destruction order compatibility issue was pointed out, and before the alternative RFC emerged.

• The coroutines eRFC (async/await syntax). This RFC was particularly high in terms of inevitability because it was the culmination of years of discussion and past experiments in Rust itself that had already made it pretty clear that stackless coroutines were the way to go, long before anyone was talking about an “ergonomics initiative”. My memory of this thread is that it was mostly a history lesson for everyone who was not familiar with those past discussions and experiments (which is also valuable in its own way; I only knew a few bits of it prior to that thread).

• The elision 2.0 threads. I particularly want to highlight this one because it never even reached an RFC, and to the best of my knowledge from spending way too much time stalking this forum and the RFCs repo, there isn’t going to be an RFC on this subject any time soon. My impression is that the thread simply didn’t reach a strong consensus and no “slam dunk” alternatives emerged in the discussion, and I guess the decision was to do nothing (for now) in favor of other issues where it was much clearer that progress could be made.

• Delegation of impls. As the roadmap issue says, this one has “stalled out”. I really want this to happen, but this is a good example of an ergonomics RFC that didn’t just get merged through sheer inevitability.

• Implied bounds. We only just got an RFC here so there’s not a lot to say yet, but so far I haven’t heard anyone say they would prefer the “do nothing” solution to even the most conservative option presented in this RFC.

• Non-lexical lifetimes. Pretty much the same as implied bounds. Does anybody not want this?

---

But I also have a non-meta comment on the module system stuff that all of this meta discussion ties into. As ergonomics issues go, the module system is unique (in my opinion) in that:

1. There’s a relatively strong consensus that something should be a lot better than it is, but apparently almost no consensus at all on what that something is.
2. The two major proposal threads we’ve seen from the lang team on this subject (I didn’t miss any, did I?) have not been backwards-compatible papercut bandaging or small new sugar-y features like the most of the other ergonomics RFCs. Rather, both of these proposals amounted to “replacing” at least a large part of today’s module system with something else and would probably require epochs/checkpoints.

I think it’s the combination of these two factors that is making these threads as bikesheddy, lengthy, non-convergent and meta as they currently are.

In particular, I currently do not believe that it is inevitable that the module system will be changing in a major, backwards-incompatible way. It might turn out that a few backwards compatible tweaks solve all the major issues and it’s just not worth doing anything more drastic.

[phaylon]

I should say first that I don’t believe in any way that anyone is arguing from a bad place. I appreciate all the work that is done, and find the ergonomics initiative a very good idea. I don’t think any of them aren’t worth it or anything like that, quite the opposite.

I believe part of the problem is that it is really hard to have a feeling about the language-future if you’re only following it down to a certain level. The match ergonomics changes were a good example for me. While the finally accepted RFC has a more conservative scope, I get the feeling that it is considered a first step. The lifetime elision discussion had talk about re-using the ref keyword, and that epochs and the match ergonomics make it available.

These can be quick brainstormed thoughts, or they could be changes the driving members are really passionate about. When you’re just following along, these things are really hard to tell. Similar effect with the dyn Trait changes. In some places I’ve heard it being talked about as a certainty. But that might have been just excitement.

As for the word “explicit,” I’m not sure I know a better one for “details specified more precisely and verified by the compiler,” which is usually my main concern.

I’m not sure what a good solution would look like. To me a solution would be for such things to be argued into #![feature(..)] mode first, and once that worked out it’s final default form is decided. Does that make sense phrasing wise? That defaulting is a separate concern to the feature?

Most of these features (match, extern crate, mod) should already have big wins even when they’re still under a stable feature flag. And even if an epoch or future version turns those features on by default, if I can #![feature(explicit_bindings, explicit_mods, explicit_deps)] I’m still happy, because I still have the possibility. My worries start when the discussion seems to be about removing those abilities. You could say I’d prefer a more capable Rust to a new Rust.

Looking back over the ergonomics parts that I worry about, they’re all about hiding details. Which can be an immense win, but personally I do often care about those details. That’s why I liked the spirit of the new lifetime elision talks, because they unhide where elision is happening.

[pcwalton]

Please no… In Rust 2012, all paths were relative. I think I am speaking for everyone when I say anyone who used Rust in 2012 remembers absolute use statements being a large improvement.

Relative use paths are a non-starter because they are literally impossible to implement. The reason why they were changed is that Rust’s name resolution system was really really broken. As in, you would add and remove imports and random other imports in totally unrelated files would spuriously fail to resolve, simply because the compiler would try to resolve things in some order and bail out based on heuristics. I added the absolute path restriction because it was the only way I could actually fix name resolution in the rewrite.

ECMAScript 6 attempted a static module system based on relative imports and similarly had to give up because it was deemed unimplementable. As I recall, in fact, I went to @dherman for help, since he was working on a similar system for ES6 at the time, and he gave me the idea of absolute imports as a quick fix.

Be careful what you wish for.

[rpjohnst]

I do not think the lang team has been ignoring the “don’t do this, it’s not worth it” perspective. Rather, I think these RFCs felt inevitable because the lang team did a good job choosing the “low-hanging fruit” in terms of possible ergonomics improvements.

I thought that was explicitly not the complaint at this point in the thread. Rather it was the “don’t do this, let’s find a better way to solve the problem.” In other words, “a few backwards compatible tweaks solve all the major issues.” Most of the team’s effort has been toward, as you mention, big flashy backwards-incompatible changes unlike the more conservative attempts by others in the thread.

Interestingly though, @withoutboats’s initial proposal is much closer to “a few backwards compatible tweaks.” It explicitly says “Of course, it needs to be backwards compatible, which means we can’t actually eliminate syntax or completely change its semantics,” and it’s basically 1) the current implicit extern crate RFC, plus 2) implicitly placing modules into the tree based on their path, rather than a mod declaration. I would love to see more work on that approach- I attempted it with my Python-inspired proposal, for example.

[Ixrec]

Could you expand a bit more on why it’s impossible to implement? I’ve somehow never heard of this problem before despite following ES6 modules and Rust.

[Ixrec]

I thought that was explicitly not the complaint at this point in the thread.

I think it was when I started writing that post and then while I was writing it you made a post that explicitly said this wasn’t the issue, but it seemed like there was at least one other person who probably still had that complaint so I kept writing it anyway although now I can no longer remember who that was. But I’m pretty sure I was responding to a thing someone said at some point. Probably.

This thread is too active.

[pcwalton]

Because when you have reexports you have no idea what a name might refer to, since you might not have resolved it yet. Cases like this were problematic:

mod a {
    pub use b::c;
    pub use c::*;
    mod b {
        use c::d; // what is "c"?
        ...
    }
    ...
}

You might think “oh, just globally iterate to a fixed point”. Yeah, we thought so too. It doesn’t work.

[arielb1]

You can do this with today’s Rust using imports with self

mod a {
    pub use self::b::c;
    pub use self::c::*;
    mod b {
        use self::c::d; // what is "c"?
        ...
    }
    ...
}

[pcwalton]

No, you can’t, because imports from parent modules are not visible in child modules.

I’m pretty unhappy that I have to defend absolute imports again. This was solved back in 2012.

[troplin]

Because when you have reexports you have no idea what a name might refer to, since you might not have resolved it yet. Cases like this were problematic:

mod a {
    pub use b::c;
    pub use c::*;
    mod b {
        use c::d; // what is "c"?
        ...
    }
    ...
}

You might think “oh, just globally iterate to a fixed point”. Yeah, we thought so too. It doesn’t work.

I think this is no problem if “relative” always means “relative to the current module”, and not “relative to the current or any parent module”.

It would just resolve to:

mod a {
    pub use ::a::b::c;
    pub use ::a::c::*;
    mod b {
        use ::a::b::c::d;
        ...
    }
    ...
}

I don’t see any problem with that. It’s what I would have expected.

[pcwalton]

Then 95% of module imports in practice will be of the form:

use super::super::super::foo::Bar;

or

use ::foo::Bar;

Not an ergonomic improvement.

[aturon]

Just a quick note: discussion has been moving extremely quickly, and I think at this point it’s best for me to spend some time writing up a fully fleshed-out blog post following up on this thread and presenting a proposal in a rather different direction. I’m gonna bow out for now.

[troplin]

Then 95% of module imports in practice will be of the form:

use super::super::super::foo::Bar;

or

use ::foo::Bar;

Not an ergonomic improvement.

IMO it is. Personally, I never use super, it smells of bad encapsulation. For absolute imports the additional :: is a rather low price and IME imports from submodules are quite frequent and those are improved a lot, you just don’t mention them.

[pcwalton]

Imports from submodules are not nearly as common in practice as imports from ancestor modules.