[Pre-RFC]: Packages as Namespaces

I feel like it would be a good idea for any proposed RFC related to squatting or namespacing to do a little research into other languages registries (PyPI, npm, RubyGems, Maven, etc.) to see if other more mature (in terms of age) registries have:

  • Experienced repeated abuses from squatting
  • Done anything about it. And was it effective?
  • Set up associations between related packages
  • Learned lessons from past successes/failures in this area

Rust is far from unique in wanting to set up an open source collection of packages, so I’d have to imagine there are some other-language registry maintainers we could learn a lot from.

5 Likes

Regarding squatting and npm, I’ve posted in the squatting thread: Crates.io squatting

Yeah it seems like we would have to introduce a new sigil which is (1) not a binary operator, and (2) not part of the current name syntax (so just not :: or .). \ would work from a technical perspective, but I agree with your aesthetic misgivings. I don’t really have any obviously better ideas, but I’ll throw a few of them into the mix:

::: would work, but I think it would make people’s eyes swim.

:> doesn’t look terrible to my eyes, ::> could also work.

I could really use help figuring out what color to paint the fence.

The problem with this approach is that it adds even more complexity to the module system for users to deal with, which is already a weak point for rust. I would hate to undo the effort put into simplifying that aspect of the language.

We could also follow npm and make prefix the namespace’s name with some sigil, like @tokio/core, which would then be parsed as a single ident.

I’ve looked around at some prior art to get an idea of how viable a single flat namespace is in the long run. npm’s “scopes” seem like the closest thing to this proposal in the wild (the npm github issue where they adopted scopes has a discussion which is relevant here). Most flat-namespace ecosystems seem to have some sort of formal arbitration policy around package ownership, with ruby being the notable exception[1].

I think there are still legitimate concerns with every permutation of this proposal that has been discussed, but I think the new-sigal variant is good enough to deserve a day in court. I’ll start writing something up. It has been really useful to get all this input during the Pre-RFC process. Thanks everyone!

[1]: Python has: https://www.python.org/dev/peps/pep-0541/, npm has: https://www.npmjs.com/policies/disputes, I struggled to find a policy for rubygems.

This is my personal opinion. It should not be taken as a statement from the crates.io team, nor does it imply that anybody else on the team shares my opinion.

In general, I like this technique, although I think we need more user research to make sure this is solving the problems we hope it’s solving. Namely, as said other places in this thread, this really only addresses the “shared branding”/officialness problem. Is this a problem worth solving? Is this the best solution to that problem?

In addition to the questions already raised about sigils/naming/etc, I’d like to see more details around permissions made explicit in an eventual RFC along these lines: it seems like whoever is designated as an owner (either directly or via team ownership) of crate “foo” should be able to create new crates “foo/whatever”. Then should crates “foo/whatever” automatically get the same owners of crate “foo”? That seems like the common case and would save folks time having to remember to add the other owners, but are there scenarios where that wouldn’t be desired? If, after creation, the owners of either the crate “foo” or the crate “foo/whatever” are changed, should those changes be synced to the other crates in the group? Only if the top level crate’s owners are changed? What would folks expect?

EDIT: Also, thank you @ethanpailes for taking the time to write this up! :heart:

6 Likes

I like this style of proposal since it gives two desirable properties at once: affiliation and namespace security. I wonder if symbol mangling could be done in a backwards compatible manner by using double underscore as a mangled separator: the crate foo/bar becomes extrn crate foo__bar;. Icky, but if nobody has a crate name with double underscores already it could be done without changes to the ecosystem.

With regard to Carol’s questions of namespace author permissions, the obvious answers are either “make comprehensive security permissions that can be twiddled to do whatever a user might want”, or “crate namespace groups should trust the members of that group and do their own governance”. I like the latter solution.

Something else that I don’t think has been brought up yet: at the risk of making a hairy issue more complicated, is there any reason the level of nesting should stop at one?

@carols10cents, the note about permissions is a really good one. I had definitely planned to flesh the permissions model out more in the full proposal. I think @icefoxen’s point about basically trusting the maintainers of a top level package to manage their own community is a good one. Selecting good defaults is probably the key thing for this RFC.

With respect to user research, I think that looking at why npm adopted scopes can provide a lot of value. They have already done some of the legwork about this sort of thing and talked about it in the open, so that’s great. I started this discussion because of the flood of chatter about this stuff on /r/rust and other boards, and I think those discussions can also be mined.

I promise I’ll write up the actual RFC real soon now.

If we did have multi-level nesting, I wonder if it could start one level up with the registry. In order words: @foo/bar would be short for @cratesio/foo/bar and registries would just be namespaces that happened to have their own URL.

Sourcegraph search of the index for __

repo:^github\.com/rust-lang/crates\.io-index$ __

One crate, teardown_tree___treap, which is

Do not use - intended for internal use of teardown_tree crate

Which seems like it would enjoy being teardown_tree/treap instead?

This is the singular occurrence of a double underscore in the index, and is indeed a triple underscore.

(This post is merely meant to be informative and not issue support one way or another)

3 Likes

Awesome datapoint!

An interesting use of namespaces would be to relax orphan rules. This RFC states that nobody can write diesel-chrono crate that integrates the two crates that would like to stay independent of each other.

However, if there was diesel:bike:chrono namespaced crate, then it would be known that all the diesel:bike:* crates are from the same author/owner/org, so they would have power to enforce their own orphan rules across their own crates.

This way the diesel crate could remain abstract and not depend on chrono, and the diesel:bike:chrono could add chrono support by being privileged to implement traits that currently only diesel can.

1 Like

As an FYI, crates.io team member here, I worked for npm for 3 years. I have been and intend to continue sharing the best practices I learned from that team. If you have any questions, feel free to direct them to me.

As for why name spaces were originally introduced, it was to help create a paid product, private packages. Very few people used them, until around 2 years later when we introduced another paid product, organizations. Free and paid organizations are the primary use case that folks use and it’s largely as a result of the increased and more ergonomic permissions controls. crates.io already affords this by letting folks leverage GitHub teams. So strictly from a npm point of view, crates.io does not currently have a motivation for this feature as we don’t need to make money and already give folks the ability to organize permissions via GitHub organizations.

3 Likes

Background: crates.io team member, npm engineer 3 years

I’d like to share that package disputes on npm are handled by a rather large team of paid support engineers. Disputes are by far the largest time burden they have. They are handled manually. I believe, through my discussions with Bundler/RubyGem maintainers that they do not have this policy because they cannot support a support team large enough to thoughtfully handle these requests.

At the moment, the crates.io team also does not have this set of support engineers. Until the team is able to grow (I have advertised joining the team multiple times in the past few weeks with literally zero people taking me up on the offer), I cannot imagine how we would be able to take on the workload of supporting this.

7 Likes

That’s a cool point about orphan rules. I think I’m not going to address it in the proposal I put forward, but if optional package based namespaces are adopted it would be a cool follow up.

@ag_dubs, that sounds like a point in favor of this idea. Hopefully optional namespaces would reduce the number of disputes that could come up and take some of the burden off the crartes.io team.

Until the team is able to grow (I have advertised joining the team multiple times in the past few weeks with literally zero people taking me up on the offer), I cannot imagine how we would be able to take on the workload of supporting this.

I'm a little confused about what you mean my this. Are you talking about the code changes required to allow crates.io to understand namespaces? I'm not proposing any sort of dispute resolution mechanism, so this proposal would not impose any sort of mandate on anyone to start playing judge.

I suggested a more aggressive name spacing using epochs here:

Best namespaces proposal so far. One addition: would be great to let owner of the “parent” namespace to add collaborators who also have right to publish/edit their crates.

Does anyone have thoughts on what to do about name mangling? I’ve been trying to figure out how the module path separator is mangled currently, but I didn’t find any quick answer.

I’m sorry for letting this thread die off. In the process of writing this idea up into an actual RFC, I became more and more concerned that this was going to add too much complexity to the language, and especially the naming system. Recent posts about rust language design being mostly finished except for a few big things that are still in the pipeline have further convinced me that making any changes to rust-the-language would be a bad idea.

I still think that packages-as-namespaces is the best approach to namespaces, but I just don’t think namespaces are needed enough that it is worth spending the complexity budget in the language itself. Packages as namespaces might still be a good idea, but it would have to be handled completely within crates.io.