The Great Module Adventure Continues

[tcr]

Also, before the meeting, it seemed like most participants were leaning towards leading-crate and leading-::, so it’s quite surprising that the meeting suddenly turned people around. Can someone do a more detailed recounting of what’s so bad about fallbacks, what transition options have been discussed and why they have been discounted as realistic options?

I’d also like to follow up on this. I’m unsure why for an epoch transition fallback is undesirable, when as far as I can tell rustfix could correct non-leading-:: extern imports to ::-leading extern imports very easily. Is there a distinction between fallback that has ambiguous interpretation between epochs, and fallback which is just preserving legacy syntax for an epoch?

After reading the leading-sigil discussion: has there been suggestion of dropping the keyword use entirely? e.g.

use crate::relative::import;
use regex::Regex;

// becomes

crate relative::import;
extern regex::Regex;

use plus sigil seems like a suboptimal exploration than leveraging keywords, if possible.

[djc]

After reading the leading-sigil discussion: has there been suggestion of dropping the keyword use entirely?

It’s an interesting idea, but I guess the cost of two new keywords would be on the high side.

[jmst]

Maybe we could go with the simplest changes that achieve the goals in the first message while preserving backward compatibility?

1. Introduce crate:: paths
2. Regarding the parsing ambiguity, treat “crate::” as a crate path if there’s no whitespace between “crate” and “::” and as a “crate” visibility modifier if there is whitespace (effectively this means treating “crate::” as a token distinct from “crate”). “crate ::” should never be parsed as a crate path even when a “crate” keyword would not be valid
3. Warn on “use foo::bar”, suggest “use crate::foo::bar” instead or “use self::foo::bar” in the crate root
4. Warn on “::foo::bar” in fn/items, suggest “crate::foo::bar” instead or “foo::bar” in the crate root
5. Warn on “std::cmp::min” in fn/items in the crate root, suggest “::std::cmp::min” instead
6. Path not found errors for relative paths should suggest to add “::” in front if that would be a valid path
7. Path not found errors for “use” should suggest to add “self::” in front if that would be a valid path
8. Make extern crate implicit depending on a Cargo.toml option set by default for new projects and implied by a new epoch
9. Allow implicit “extern crate” names to alias crate-internal items. The extern crate would be used with absolute paths, and of course the item would be used with “crate::” paths
10. Warnings can be enabled in a new epoch only if considered too noisy otherwise. It’s also possible to make them error in a new epoch as well.

This is essentially “leading crate” with “fallback” (assuming that “fallback” means continuing to support the current syntax), and I disagree that it has the “potential for confusion” since it’s usually pretty clear whether a path is external or internal and the “unclear” paths produce warnings anyway.

[rpjohnst]

2. Regarding the parsing ambiguity, treat “crate::” as a crate path if there’s no whitespace between “crate” and “::” and as a “crate” visibility modifier if there is whitespace (effectively this means treating “crate::” as a token distinct from “crate”). “crate ::” should never be parsed as a crate path even when a “crate” keyword would not be valid

IMO the existing solution is simpler: treat crate in a path like any other crate name, so you always write ::crate::foo in the cases where it would be ambiguous (plus several other cases).

This is essentially “leading crate” with “fallback” (assuming that “fallback” means continuing to support the current syntax)

Yes, I actually think framing this as a bunch of warnings that could later be turned into errors is a great way to demonstrate why I like leading-crate so much. Your description works just as well for the flag-day approach, if it’s applied only to the current epoch.

One aspect that I’d like to pick out, because I don’t think it’s been discussed, is the idea of allowing implicit extern crate in the current epoch via a flag rather than via a fallback mechanism. This might be seen as opening the floodgates to lots of feature-specific flags outside of epochs, but it does simplify both the implementation and the porting work, so it might be justified in this case.

That is, because the alternatives here introduce so much churn, and the fallback is so complex, it might be worth using this approach instead of “flag day” or “fallback”:

• Stabilize leading-crate paths in the current epoch.
• Warn on paths that do not begin with ::crate/::some_dep; add suggestions along the lines proposed by @jmst both for these warnings and for path-not-found errors.
• Offer a feature flag, still in the current epoch, to make extern crate optional. Force this flag on in the next epoch.

[atagunov]

Some alternatives to ::crate in a project named xyz:

• ::xyz to address own lib crate from within itself
• ::bin to address own bin crate from within itself
• ::tests to address own tests crate from within itself

[mark-i-m]

I haven’t followed this thread too closely so far because it’s really huge, but trying to catch up… here is my 2 cents:

• I vastly prefer keywords to sigils. This may be partly due to my aesthetic tastes, though… I have used C++ alot before, and I have been using Rust for a couple of years now, and I still find paths starting with :: to be a huge eyesore.
• I don’t see the verbosity of extern as a problem. IMHO, if you find your self using an absolute path for an extern inline often, you are probably doing something wrong. Usually, if I have to use paths inline, I use a relative path. And if use something often enough, I use it. Thus, leading extern seems to give 1path without too much verbosity (EDIT: I used the word “churn”; I meant to write “verbosity”). Does anyone have numbers on how common inline absolute paths are?

How about new path syntax a.b.c?

use std.io; use crate.module;

1path, no fallback.

I like it, but it’s ambiguous with struct field access, right?

[nikomatsakis]

Sorry, still haven’t carved out time for a detailed response, but I want to reiterate that it’s worth giving features a spin for a while to see how they feel. I’ve been using extern paths (along with a bunch of other features) in some of my side-projects, and I have to say that they feel pretty nice to me (though the other options also did). Verbose but very clear. Just add this to your project:

#![feature(
    crate_in_paths, 
    decl_macro, 
    extern_in_paths,
    crate_visibility_modifier,
)]

Then you can remove the extern crates and convert to use extern::{...}. Technically, you don’t need the last one, it allows you to do crate fn instead of pub(crate) fn – but I think it’s good for a more rounded picture. You do want decl_macro, though, so that you can import macros into scope, since there is no extern crate to #[macro_use].

I would give yourself a day or so, personally.

[josh]

Is there any combination of features, or any proposal on the table, that would work like this?

use extern::rayon::prelude::*; // prelude from the rayon crate
use ::helpers::foo; // foo from the helpers module of the current crate

That seems like an unambiguous syntax, one that supports 1path, and one that doesn’t make local crate paths excessively long.

[nikomatsakis]

I suppose you can get that by just … not using the crate_in_paths feature.

[josh]

Will that produce consistent :: paths that work everywhere, including in use statements and in function bodies, in any submodule of a crate?

[nikomatsakis]

Will that produce consistent :: paths that work everywhere, including in use statements and in function bodies, in any submodule of a crate?

Seems like it – I mean ::foo paths already work anywhere. The main downside is that you have to write use ::foo::bar to reference things from your own crate, and some consider that unsightly. On the other hand, if you do:

use ::{
    foo::bar;
};

use extern::{
    std::cmp,
};

then maybe it’s fine.

One thing I do like about this proposal is that it is basically summarized as:

1. extern crate can now be elided in favor of an extern path
2. we now write the leading :: in a use, for consistency

which is a clean delta on the existing system.

[josh]

That’s exactly my feeling about it, as well, and thanks for summarizing it. And while I understand that extern feels long when looked at on its own, I feel like this just makes it possible to change longer things to shorter things. You can change extern crate foo; to just use extern::foo;. You can combine that with using a symbol from it: use extern::foo::symbol; (or use extern::foo::{self,symbol}; if you still want foo itself). And if you really want to, you can change the combination of extern crate foo; at the top and foo::symbol in code into just extern::foo::symbol in code. But all of those are options, and none of them makes code longer. (Note, for instance, that in that last case you could also keep using use extern::foo; and then foo::symbol.)

Also, I feel like having to write ::foo::bar in a use statement is about the smallest change we can make that provides 1path consistency with writing the same ::foo::bar in arbitrary code, so that seems worth it for that consistency.

[rpjohnst]

You can change extern crate foo; to just use extern::foo;.

The thing that makes leading-extern feel long is the repetition. extern crate foo is only written once (or not at all with any of the alternatives), and then you have use foo::symbol or ::foo::symbol many times, everywhere it’s used. extern::foo::symbol takes the extern crate statement and broadcasts it out everywhere. Even if it becomes idiomatic to import things in blocks that’s still a lot of repetition.

[jmst]

I think “extern::” might be acceptable in use statements (although there’s still quite an impact if one uses multiple use statements as opposed to putting everything in a single use extern::{}), but it seems way too verbose when using items without importing them.

[Xanewok]

The thing that makes leading-extern feel long is the repetition. extern crate foo is only written once (or not at all with any of the alternatives), and then you have use foo::symbol or ::foo::symbol many times, everywhere it’s used. extern::foo::symbol takes the extern crate statement and broadcasts it out everywhere. Even if it becomes idiomatic to import things in blocks that’s still a lot of repetition.

I may be lacking some context to the path changes, but doesn’t use extern::foo effectively pull the whole foo::{...} crate namespace into the local scope, which means you can just follow it up with subsequent use foo::{symbol, ...} (that is, without having to prepend it with extern::foo::)?

[josh]

Previously, you couldn’t use extern crate items without importing them at all.

Also, even if you don’t group all extern modules together in one use block, grouping all imports from a given crate together suffices to make this strictly less verbose than the alternative.

[nikomatsakis]

I think what josh is saying is that use extern::regex; at the root level acts exactly like extern crate regex; today (*) – and, if we don’t have crate::foo paths, then so does ::regex::Regex or what have you. This relies on the fact that use statements are really items, which is admittedly not my favorite feature (**).

(*) modulo macros

(**) I regret acceding to it, seems obviously like a case where a “beautiful underlying model” can lead to a confusing user experience.

[johnthagen]

Can we learn anything from the C++20 modules spec that is under draft (whether good or bad)?

The basic draft seems to look like:

add these two keywords to Table 3 in paragraph 2.11/1: module and import.

import std.io; // make names from std.io available
module M; // declare module M
export import std.random; // import and export names from std.random
export struct Point { // define and export Point
    int x;
    int y;
};

[rpjohnst]

I may be lacking some context to the path changes, but doesn’t use extern::foo effectively pull the whole foo::{...} crate namespace into the local scope, which means you can just follow it up with subsequent use foo::{symbol, ...} (that is, without having to prepend it with extern::foo::)?

It does pull foo into the local scope, but that doesn’t enable use foo::{symbol, ...} because use takes absolute paths. (You can write use self::foo::{symbol, ...}, though.)

The repetition I’m talking about is across files, not within them. Instead of a single extern crate foo in lib.rs, you now have many extern::s prepended to all the use blocks/statements in all the files that previously just had use foo::.

[aturon]

I’ve been turning over @josh’s idea in my head, and am finding it pretty appealing. Here’s my preferred variant:

• Allow paths to begin with extern followed by a crate name.
• Encourage use of nested import groups.
• Deprecate use statements that do not start with ::, extern, super or self.
• Add std to the prelude, rather than just injecting it into the crate root.

Upsides:

• Provides “1path” semantics: non-deprecated use paths are valid paths with the same meaning everywhere.
• Eliminates all special-casing of the root module, including std injection.
• “Qualified” paths (i.e. those that are valid in use) have a very straightforward syntax and semantics; confusion seems very unlikely.
• Intra-crate paths are more concise than in most other variants (leading ::rather than leading crate::)
• No fallback required.
• Minimal delta over today’s module system, with a very clear mental model for the transition (“we replaced extern crate with the concept of extern paths”).
• Today’s paths can safely coexist with these paths, at worst triggering a lint. Code from StackOverflow will continue to work.

Downsides:

• extern is relatively verbose, especially for “in-line” uses within expressions.
  • Mitigation: usage of nested import groups
  • Mitigation: std being in the prelude, which accounts for a large portion of “in-line” extern usage.
• Leading :: means that many use statements will look a bit sigil-heavy.
  • Mitigation: the fact that the sigil is ::, the path separator, both makes the syntax stand out less, and provides a very “obvious” meaning in terms of “go to the root”. It may feel a bit noisier, but it’s unlikely to be viewed as a “shark-jumping” moment for Rust.

The repetition I’m talking about is across files, not within them. Instead of a single extern crate foo in lib.rs, you now have many extern::s prepended to all the use blocks/statements in all the files that previously just had use foo::.

FWIW, I don’t see this as such a bad thing: when I’m navigating an unfamiliar codebase, I often find it very helpful to have a clear split between imports from external crates and those from within the crate. That’s usually because the public interface of external crates is relatively well-abstracted (so easy to get a quick mental model on), while the internal module structure of a crate is a bit more arbitrary.

People already make this separation informally by spacing out the imports. Here we’d be codifying that by grouping the external imports into an explicit use extern::{ ... } block, which doesn’t seem so repetitive to me.

Also, doesn’t the same argument apply to leading crate with respect to internal paths?