Pre-RFC: (hyper)minimalist namespaces on crates.io

[bascule]

I apologize to everyone in advance for "yet another namespace thread", but after some recent discussion on this subject, this approach popped into my head and I think it's simple and clever enough that it might just work.

Goal: support promoting an existing crate to a "namespace" by blocking new package registrations for crates whose names begin with a "namespace" prefix, as a purely crates.io-side change.

1. Given a published crate/package foobarbaz, if no other foobarbaz*-prefixed crates have ever been published, allow a user to opt into converting foobarbaz into a "namespace".
2. The net effect of converting a crate to a "namespace" is to block publishing of new packages in the "namespace" by anyone but the owners of the namespace, i.e. only the owners of crate/namespace foobarbaz can publish new crates with names prefixed by foobarbaz*
3. There is no third step, or rather: newly published crates in the foobarbaz* namespace are otherwise just crates. Their owners work like any other newly published crate: the publisher is the initial owner, and they can add whomever they like.

If a feature like this were to be adopted, clearly there's a lot more that could be done as secondary steps. It'd be great to brand crates in a namespace with the namespace they belong to on crates.io in some way. It'd be great if Cargo reflected it somehow. Perhaps there could be an option to have new crates published in a namespace to inherit the owners of the namespace. But that can all be done as follow-ups.

[josh]

I personally think this could be a good idea in principle, but not in practice.

One potential issue: does anyone get to judge whether the namespace is "too generic", or could people claim anything starting with an extremely generic prefix?

[bascule]

I think there are some additional restrictions that could be placed on what names are allowable for a namespace. A minimum length certainly seems like a good one (5 perhaps?)

[steveklabnik]

What happens when I publish a crate named “a” and make it a namespace?

[bascule]

I'm (not really) guessing there are a few crates that begin with a, so that name wouldn't be available as a namespace. But that sort of problem is a good argument for a minimum length.

[steffahn]

With such a mechanism, a finite amount of "namespaces" could eat up all of the crate names that are left. Admitted it is not a very small finite number but even the possibility to squat huge chunks of possible names seems super suboptimal.

For example, as a minimal length 5 was mentioned, there are only roughly 10 million length 5 a-z words. All the as-of-yet unused ones could be taken quickly. Add to that for every existing crate name, like "crossbeam" all the prefixes of length >=5 plus an extra letter, like "crossa", "crossc", "crossd",..., "crossba", "crossbb",... (leaving out the ones that are already prefix of any existing crate) and you can indeed cover all the remaining possible names with roughly 26*(average length of crates.io names)*(number of crates already published) namespaces.

Conclusion: IMO this finiteness contradicts the idea that Rust might still become many orders of magnitudes more popular than today and might be around for the next 100+ years.

[bascule]

@steffahn

Add to that for every existing crate name, like "crossbeam" all the prefixes of length >=5 plus an extra letter, like "crossa", "crossc", "crossd",..., "crossba", "crossbb"

You couldn't claim crossbeam as a "namespace", because:

if no other foobarbaz* -prefixed crates have ever been published

...doesn't hold. There are already many crossbeam*-prefixed crates published. So part of the art of claiming a "namespace" under this model is finding a name which is already a unique prefix within the existing ecosystem of published crates.

(Sidebar: perhaps there could be a mechanism for someone who owns foobarbaz along with all of the other other existing foobarbaz* crates to retroactively claim a "namespace", but that feels like a follow-up discussion)

[samsieber]

A namespace of foobarbaz blocking only foobarbaz-* instead of foobarbaz* would make it a little less worrisome. There are still only a finite number of words though.

[CAD97]

I think this is the same proposal as the previous "packages as namespaces" proposal.

I still like it in theory, but I'm honestly not too sure in practice. (And I still think multiple lib crates in a package serves the "branding" need for namespaces better, but I acquiesce that the crates team wants to maintain package=lib.)

[bascule]

@CAD97

I think this is the same proposal as the previous "packages as namespaces" proposal.

Not quite, although I thought about mentioning it. This scheme is inspired by that proposal in part, but is a conceptual simplification.

To highlight some of the differences, let me quote parts of that proposal and describe how this one is different:

[Pre-RFC]: Packages as Namespaces

There’s been a lot of chatter about adding namespaces to crates.io recently, so apologies if this idea has already been discussed. Summary Allow packages to be used as namespaces in a way that is backwards compatible with the existing flat namespace. Details The owners of a crate on crates.io can add sub-crates which are referred to by the name ‘toplevelcrate/subcrate’ in Cargo.toml files. For example, instead of publishing regex-syntax, the regex maintainers could have published regex/syntax.…

The owners of a crate on crates.io can add sub-crates which are referred to by the name ‘toplevelcrate/subcrate’ in Cargo.toml files

My proposal:

• Does not require any sort of "linking" (although it might be nice to model which crates belong in a namespace in crates.io's database). In fact, by the rules of this scheme retroactively linking crates is impossible, because if they existed and fit the rules, they would prevent the namespace from being created in the first place.
• Does not provide a way to associate existing crates with a "namespace". Instead, it only impacts newly published crates in a namespace.
• Does not introduce new syntax in package names i.e. namespace/subpackage. Instead it provides a rule for a prefix string in package names.

[CAD97]

Ah.

I know I saw a proposal that was very similar previously, though.

(Roughly if you own package foo and nobody else owns a foo-* crate, you can claim all foo-* crates as reserved.)

[steffahn]

Maybe you're thinking of this comment from OP themself

Pre-RFC: User namespaces on crates.io

One solution for this which has been proposed by several people, which is quite different from the solution proposed in this pre-RFC, is adding namespacing purely on the crates.io side (or an alternative registry's side) in a way that doesn't require any changes to the Rust module system or even its knowledge/awareness of namespaces (i.e. a registry-side-only change). This could be accomplished by letting someone claim the foobarbaz-* / foobarbaz_*-prefixed namespace if and only if there are n…

Also, as far as I can tell, the additional - (i.e. you only claim namespace-* instead of namespace*) solves the “a finite number of namespaces can reserve every name that's left is” issue that I tried to explain above.

[withoutboats]

@bascule

• Does not introduce new syntax in package names i.e. namespace/subpackage . Instead it provides a rule for a prefix string in package names.

I wonder if we could elaborate all of the costs of this aspect of the other proposal. I know it makes the proposal cross cutting, which you've highlighted as a problem, but it seems like a fairly narrow change to cargo that would be manageable.

I note that the notion of linking and associating existing crates with a namespace are not necessary features of the other proposal: you could allow every crate to become a namespace using a newly introduced separator without allowing moving existing crates between namespaces.

The big disadvantage of this proposal is that existing projects (crossbeam, tokio, etc) would not be able to use this feature. This seems really detrimental to me. But I think a version in which every crate can be a namespace could be stripped down to an MVP that does not introduce very much complexity. Maybe I'm wrong.

[HeroicKatora]

@bascule

Does not introduce new syntax in package names i.e. namespace/subpackage . Instead it provides a rule for a prefix string in package names.

@withoutboats

I wonder if we could elaborate all of the costs of this aspect of the other proposal. I know it makes the proposal cross cutting, which you've highlighted as a problem, but it seems like a fairly narrow change to cargo that would be manageable.

That's a step in a right direction. I was wondering if it were possible/ergonomic to solve the name shortening problem in cargo instead of the crate registry, and if that reduces or increases some of the costs. The pain of long names is that you often type the same prefix multiple times when you use multiple crates from the same group. Without regards to there already being actix- prefixed packages, this may look like so:

[dependencies-group.actix]
# Maps to a dependency on actix-web
web = "3.0"
# Maps to a dependency on actix-rg
rt = "1.1"

[Manishearth]

I think foobarbaz-* would work, but foobarbaz does not.

I do not think the costs of foo/bar are significant, as @withoutboats alludes to. I'm much more in favor of packages as namespaces.

[kornel]

I suggest doing this with slight tweaks:

• owner of foo can reserve foo-*.
  • The dash naturally matches current practice.
  • Prevents namespace exhaustion, because foo-* doesn't squat on foo2-*.
• reservation affects only creation of new crates (i.e. check for namespace ownership is done when cargo publish is run for the first time. After that it's just a regular crate, and cargo publish continues to use actual per-crate ownership information)
  • This allows existing projects to reserve their name prefix for future use without having to fight over existing crates.
  • It's very flexible, and it's probably the smallest/easiest way to implement it.

Such approach has been suggested a few times, and I think that's the most realistic of all proposals. It's fully backwards compatible with existing language, tooling and most importantly, crate ecosystem.

The current "global" namespace of crates has all the crates that every Rust project relies on. Rust has to keep all of them working exactly as they are, forever, to keep its stability and compatibility promises. This means that any proposal that introduces new crate name syntax would fragment crate names into new-namespaced and legacy-global names. Prefix reservation is a neat solution that makes new namespaces look exactly like old name prefixes, so there's no obvious spit in the ecosystem.

[bascule]

@kornel I like those suggestions

[josh]

@Manishearth

I do not think the costs of foo/bar are significant

Likewise. That would not be hard to support in Cargo, or other tooling. We'd need a convention like "/ translates to _ in crate identifiers by default", and an appropriate encoding scheme for the crate index, but otherwise, that's not hard to support. And it'd mean we could do "every crate is a namespace" very easily, without any potential conflicts.

[bascule]

@Manishearth

I do not think the costs of foo/bar are significant, as @withoutboats alludes to. I'm much more in favor of packages as namespaces.

Another potential syntax to consider is using . as the separator, if that's possible.

It would map more naturally to TOML, where the . separator allows you to conveniently define nested tables.

Example:

Rust Playground

[dependencies]
namespace-a.package-a = "0.1.0"
namespace-b.package-a = "0.2.0"
namespace-a.package-b = "0.3.0"
namespace-b.package-b = "0.4.0"

...parses to the following toml::Value:

Table(
    {
        "dependencies": Table(
            {
                "namespace-a": Table(
                    {
                        "package-a": String(
                            "0.1.0",
                        ),
                        "package-b": String(
                            "0.3.0",
                        ),
                    },
                ),
                "namespace-b": Table(
                    {
                        "package-a": String(
                            "0.2.0",
                        ),
                        "package-b": String(
                            "0.4.0",
                        ),
                    },
                ),
            },
        ),
    },
}

...which you can also write as:

[dependencies.namespace-a]
package-a = "0.1.0"
package-b = "0.3.0"

[dependencies.namespace-b]
package-a = "0.2.0"
package-b = "0.4.0"

[Nemo157]

Using the . separator like that would make declaring a dependency on the root of the namespace quite weird.

[dependencies]
serde.version = "1" # is this the crate serde or serde.version
serde.json = "1"