Pre-RFC: Packages as Optional Namespaces

[steffahn]

@dhardy

The downside is the potential for confusing errors when foo::bar is not a dependency:

Isn't this just about as confusing as forgetting to enable a feature already is? If subcrates are featured prominently on crates.io and in the docs of a crate (probably best by some new rustdoc feature so you can list subcrates in your main source file that get presented in the generated docs right next to the crates's top level modules for easy discoverability) they ought to be at least as easy to discover as features are currently, right?

[kornel]

How about using workspaces as a namespace?

If you own a crate named foo, you can make it a workspace, and publish the entire workspace at once with all the sub crates in it. Then / in the crate name/path would look into the workspace.

There are many projects that already use workspaces and monorepos, and when they publish, it's a chore to publish all of the individual crates. Lots of crates have -derive and -core versions. This is a pain to maintain, so e.g. tokio switched from tokio-foo to just one crate with feature flags. But if crates-io supported workspaces, then workspace == namespace would solve two problems at once, and it'd be a novel useful functionality, not just a fancy mapping of slashes to underscores.

[Eh2406]

@dhardy

I really like this proposal. For the most part the name conflict isn't even an issue: the owners of foo know that foo::bar is a sub-package, and therefore if the bar submodule is shadowed it doesn't matter.

Here is the problem I see. As of 1.0.0 I have a crate foo, with important functionality in the bar module. Now bar is getting more complicated and can be split out. For 2.0.0 I want to make a foo/bar crate, and remove the foo::bar module. During the upgrade period a user depends on foo="1.0" and "foo/bar"="2.0" and has use foo::bar in their code. rustc needs to have a "ambiguous use statement error", and so this RFC has changes for rustc.

[Manishearth]

@kornel

How about using workspaces as a namespace?

If you own a crate named foo , you can make it a workspace, and publish the entire workspace at once with all the sub crates in it. Then / in the crate name/path would look into the workspace.

There are many projects that already use workspaces and monorepos, and when they publish, it's a chore to publish all of the individual crates. Lots of crates have -derive and -core versions. This is a pain to maintain, so e.g. tokio switched from tokio-foo to just one crate with feature flags. But if crates-io supported workspaces, then workspace == namespace would solve two problems at once, and it'd be a novel useful functionality, not just a fancy mapping of slashes to underscores.

I dislike this, because it forces you to organize your code a certain way to be able to get namespaces. This is something I hate from C++, for example: C++ forces you to organize your code a certain way if you wish to use inlining and templates. One should not be forced to use a monorepo to get namespaces.

I am very much looking forward to proposals that allow you to use workspaces to co-version projects as a single "package" containing multiple co-versioned "crates": I know someone working on such a proposal. I do not think this proposal should do this, I consider it to be orthogonal.

@Eh2406

Here is the problem I see. As of 1.0.0 I have a crate foo , with important functionality in the bar module. Now bar is getting more complicated and can be split out. For 2.0.0 I want to make a foo/bar crate, and remove the foo::bar module. During the upgrade period a user depends on foo="1.0" and "foo/bar"="2.0" and has use foo::bar in their code. rustc needs to have a "ambiguous use statement error", and so this RFC has changes for rustc.

Actually, there would be no such error, foo::bar would default to the namespaced crate if both exist (foo::crate::bar is how you get to the module), and an informal contract we ask people to maintain is that if namespaced foo::bar exists, mod bar in foo should either not exist or be a simple reexport.

However, this still needs rustc changes because rustc needs to be able to handle crates with :: in their name and understand what that means.

[Manishearth]

@XAMPPRocky

The limitation requiring all namespace names to be whole words seems counter intuitive

...

I think we would be in a bad situation if not supporting hyphenated namespaces meant organisations started concatenating their names to a single word (e.g. asyncstd ) just to have namespaces that match their branding.

There is no such limitation? All valid crate names are valid namespace names. I don't see where you get this from.

[XAMPPRocky]

@Manishearth

In other cases, the "namespace root" foo may be owned by a different set of people than the foo-bar crates, and folks may need to negotiate ( async-std has this problem, it manages async-foo crates but the root async crate is taken by someone else).

You’re right, there isn’t, consider it withdrawn. I think I was confused by lack of hyphenated examples and the above sentence talking about async-std wanting the async namespace.

[Manishearth]

@XAMPPRocky

You’re right, there isn’t, consider it withdrawn. I think I was confused by lack of hyphenated examples and the above sentence talking about async-std wanting the async namespace.

Yeah, that comes from my impression that the "informal namespace" async-std seems to be occupying is async-* not async-std-*. There are non-async-std crates in that "informal namespace" (as there are under serde-*), but most of the crates published by that organization follow async-*, and I figure they'd want async/

[cuviper]

The fact that it's a keyword makes async a bad namespace example.

[Manishearth]

Well, it is, but it's not one of the original ones, so crates.io allows the name, and there is a crate squatted there which they could ask to take over. But yes, not a great example.

[ehiggs]

Is there a reason to not use reverse-DNS notation for the package names? "." is not currently a valid crate name character so it can be treated as 'reserved'. e.g. com.gitlab.ehiggs.rust-gst.

For backwards compatibility I think it makes sense to infer any non-reverse-DNS crates to be io.crates.<crate-name>.

[jmjoy]

I think the only thing we need is to lock the crate name prefix in crates.io, such as tokio and tokio-, to forbidden the other unofficial people using this prefix.

[Manishearth]

@ehiggs

Is there a reason to not use reverse-DNS notation for the package names? "." is not currently a valid crate name character so it can be treated as 'reserved'. e.g. com.gitlab.ehiggs.rust-gst .

DNS isn't stable, it changes hands, and verification is annoying. Furthermore having package names that long is kind of painful, I always hated that in Java.

@jmjoy

I think the only thing we need is to lock the crate name prefix in crates.io, such as tokio and tokio- , to forbidden the other unofficial people using this prefix.

Yes, this could be done, except we already have crates where foo and foo-bar are owned by different groups, and as mentioned in the pre-RFC, sometimes that is intentional.

[ehiggs]

DNS isn't stable, it changes hands

AIUI you don't register a domain, you register a prefix on the Maven Central servers. This is why you can have com.github.manishearth even though you don't own github.com.

verification is annoying.

But verification and signing are cornerstones of managing packages aren't they? What code am I importing into my project? Was it middleman and injected? Is tokio-naughty a project that is a trusted as tokio-core? How can we do namespacing without managing verification?

@Manishearth

Furthermore having package names that long is kind of painful, I always hated that in Java.

This is something the tooling handles so its surprising that this would cause pain. An alternative is C#'s [Organization].[Technology].[Feature]

[pietroalbini]

@ehiggs

AIUI you don't register a domain, you register a prefix on the Maven Central servers. This is why you can have com.github.manishearth even though you don't own github.com .

Just a note: GitHub, GitLab and BitBucket all allow renaming your user account, and other users can claim your old usernames.

[skade]

@Manishearth

Yeah, that comes from my impression that the "informal namespace" async-std seems to be occupying is async-* not async-std-* . There are non-async-std crates in that "informal namespace" (as there are under serde-* ), but most of the crates published by that organization follow async-* , and I figure they'd want async/

As some clarification here: async-std doesn't claim the async namespace (the naming background is "an async version of std" and not "the async.rs projects std" and tries to make sure most async- crates published are only based on futures and generally usable. It would not be named async, even if the name were available.

So it is indeed a good example to talk about there, because it is an interesting configuration to talk about.

As someone involved in the naming, I am retroactively much more annoyed that we picked async.rs as the projects name than calling async-std the name it has.

[withoutboats]

It's not at all clear to me how you'd be proposing to use DNS in package namespacing. Can you be more concrete?

[ehiggs]

I am not proposing to use DNS in package name spacing. I am proposing reverse DNS notation. This is just to have a fully qualified namespace for the packages which can often be based on the domain.

Basically what I am proposing is what Java has. It's not novel or exciting but it works extremely well and scales very well and covers a lot of the issues that need to be solve: provenance, and security.

I think the confusion is that people might believe that there is a solid link between DNS and the reverse DNS notation being used. In reality, the OSSRH process involves manual labor where accounts need to be approved. The approval process involves having someone check that you have e.g. an email account on the domain. Here is a nice video explaining the process.

[bascule]

@ehiggs

I think the confusion is that people might believe that there is a solid link between DNS and the reverse DNS notation being used. In reality, the OSSRH process involves manual labor where accounts need to be approved . The approval process involves having someone check that you have e.g. an email account on the domain. Here is a nice video explaining the process.

There was some discussion about OSSRH on another thread. As the crates.io team doesn't have a full-time staff to curate namespaces like Sonatype does, an approach which needs something like a JIRA ticket queue and manual review of each namespace is probably not going to work.

[Manishearth]

@ehiggs

AIUI you don't register a domain, you register a prefix on the Maven Central servers. This is why you can have com.github.manishearth even though you don't own github.com .

Yes, but this is worse, because this forces everyone to register on their domain or github username.

Let's say that I am perfectly fine with having my projects be unnamespaced (which I am). I'm still going to have to scramble to register net.manishearth and com.github.manishearth so that others don't use it. Projects wishing to have a project identity will likely have to register a domain to cement their identity. Sure, they don't have to, but they'll likely want to.

So you either need verification (and thus force people to buy domains), or you don't (and force people to proactively squat their own usernames and domains, and still force people to buy domains when they want a nice project identity).

The point of my proposal is that it's optional -- if you don't want to use namespacing you don't really have to think about it, and that's why it's entirely tied to existing project names -- the namespaces do not derive from anything external to crates.io.

@ehiggs

But verification and signing are cornerstones of managing packages aren't they? What code am I importing into my project? Was it middleman and injected? Is tokio-naughty a project that is a trusted as tokio-core ? How can we do namespacing without managing verification?

My proposal already solves this, directly at the source. tokio-naughty is not trusted, tokio/naughty is. We do not need additional verification tied to a whole other ecosystem (DNS) to make this work. It is easy to understand ownership as being tied to other packages.

How do you know that rs.tokio.foo is actually a tokio package in this world? What if the tokio project did not actually register that domain?

@ehiggs

This is something the tooling handles so its surprising that this would cause pain. An alternative is C#'s [Organization].[Technology].[Feature]

I've written a fair amount of Java without an IDE. I've written almost all my Rust without an IDE. Java is far more inclined to make choices that require tooling support (and is clunky without it).

[Manishearth]

@ehiggs

I think the confusion is that people might believe that there is a solid link between DNS and the reverse DNS notation being used. In reality, the OSSRH process involves manual labor where accounts need to be approved . The approval process involves having someone check that you have e.g. an email account on the domain. Here is a nice video explaining the process.

The understaffed crates.io team has been historically very reluctant to any kind of manual verification. I do not speak for them here, but I think this proposal is a non-starter for that reason alone.