Pre-RFC: (hyper)minimalist namespaces on crates.io

I'm (not really) guessing there are a few crates that begin with a, so that name wouldn't be available as a namespace. But that sort of problem is a good argument for a minimum length.

With such a mechanism, a finite amount of "namespaces" could eat up all of the crate names that are left. Admitted it is not a very small finite number but even the possibility to squat huge chunks of possible names seems super suboptimal.

For example, as a minimal length 5 was mentioned, there are only roughly 10 million length 5 a-z words. All the as-of-yet unused ones could be taken quickly. Add to that for every existing crate name, like "crossbeam" all the prefixes of length >=5 plus an extra letter, like "crossa", "crossc", "crossd",..., "crossba", "crossbb",... (leaving out the ones that are already prefix of any existing crate) and you can indeed cover all the remaining possible names with roughly 26*(average length of crates.io names)*(number of crates already published) namespaces.

Conclusion: IMO this finiteness contradicts the idea that Rust might still become many orders of magnitudes more popular than today and might be around for the next 100+ years.

7 Likes

You couldn't claim crossbeam as a "namespace", because:

if no other foobarbaz* -prefixed crates have ever been published

...doesn't hold. There are already many crossbeam*-prefixed crates published. So part of the art of claiming a "namespace" under this model is finding a name which is already a unique prefix within the existing ecosystem of published crates.

(Sidebar: perhaps there could be a mechanism for someone who owns foobarbaz along with all of the other other existing foobarbaz* crates to retroactively claim a "namespace", but that feels like a follow-up discussion)

A namespace of foobarbaz blocking only foobarbaz-* instead of foobarbaz* would make it a little less worrisome. There are still only a finite number of words though.

16 Likes

I think this is the same proposal as the previous "packages as namespaces" proposal.

I still like it in theory, but I'm honestly not too sure in practice. (And I still think multiple lib crates in a package serves the "branding" need for namespaces better, but I acquiesce that the crates team wants to maintain package=lib.)

Not quite, although I thought about mentioning it. This scheme is inspired by that proposal in part, but is a conceptual simplification.

To highlight some of the differences, let me quote parts of that proposal and describe how this one is different:

The owners of a crate on crates.io can add sub-crates which are referred to by the name ‘toplevelcrate/subcrate’ in Cargo.toml files

My proposal:

  • Does not require any sort of "linking" (although it might be nice to model which crates belong in a namespace in crates.io's database). In fact, by the rules of this scheme retroactively linking crates is impossible, because if they existed and fit the rules, they would prevent the namespace from being created in the first place.
  • Does not provide a way to associate existing crates with a "namespace". Instead, it only impacts newly published crates in a namespace.
  • Does not introduce new syntax in package names i.e. namespace/subpackage. Instead it provides a rule for a prefix string in package names.
2 Likes

Ah.

I know I saw a proposal that was very similar previously, though.

(Roughly if you own package foo and nobody else owns a foo-* crate, you can claim all foo-* crates as reserved.)

Maybe you're thinking of this comment from OP themself

Also, as far as I can tell, the additional - (i.e. you only claim namespace-* instead of namespace*) solves the “a finite number of namespaces can reserve every name that's left is” issue that I tried to explain above.

I wonder if we could elaborate all of the costs of this aspect of the other proposal. I know it makes the proposal cross cutting, which you've highlighted as a problem, but it seems like a fairly narrow change to cargo that would be manageable.

I note that the notion of linking and associating existing crates with a namespace are not necessary features of the other proposal: you could allow every crate to become a namespace using a newly introduced separator without allowing moving existing crates between namespaces.

The big disadvantage of this proposal is that existing projects (crossbeam, tokio, etc) would not be able to use this feature. This seems really detrimental to me. But I think a version in which every crate can be a namespace could be stripped down to an MVP that does not introduce very much complexity. Maybe I'm wrong.

2 Likes

That's a step in a right direction. I was wondering if it were possible/ergonomic to solve the name shortening problem in cargo instead of the crate registry, and if that reduces or increases some of the costs. The pain of long names is that you often type the same prefix multiple times when you use multiple crates from the same group. Without regards to there already being actix- prefixed packages, this may look like so:

[dependencies-group.actix]
# Maps to a dependency on actix-web
web = "3.0"
# Maps to a dependency on actix-rg
rt = "1.1"

I think foobarbaz-* would work, but foobarbaz does not.

I do not think the costs of foo/bar are significant, as @withoutboats alludes to. I'm much more in favor of packages as namespaces.

3 Likes

I suggest doing this with slight tweaks:

  • owner of foo can reserve foo-*.
    • The dash naturally matches current practice.
    • Prevents namespace exhaustion, because foo-* doesn't squat on foo2-*.
  • reservation affects only creation of new crates (i.e. check for namespace ownership is done when cargo publish is run for the first time. After that it's just a regular crate, and cargo publish continues to use actual per-crate ownership information)
    • This allows existing projects to reserve their name prefix for future use without having to fight over existing crates.
    • It's very flexible, and it's probably the smallest/easiest way to implement it.

Such approach has been suggested a few times, and I think that's the most realistic of all proposals. It's fully backwards compatible with existing language, tooling and most importantly, crate ecosystem.

The current "global" namespace of crates has all the crates that every Rust project relies on. Rust has to keep all of them working exactly as they are, forever, to keep its stability and compatibility promises. This means that any proposal that introduces new crate name syntax would fragment crate names into new-namespaced and legacy-global names. Prefix reservation is a neat solution that makes new namespaces look exactly like old name prefixes, so there's no obvious spit in the ecosystem.

11 Likes

@kornel I like those suggestions

Likewise. That would not be hard to support in Cargo, or other tooling. We'd need a convention like "/ translates to _ in crate identifiers by default", and an appropriate encoding scheme for the crate index, but otherwise, that's not hard to support. And it'd mean we could do "every crate is a namespace" very easily, without any potential conflicts.

5 Likes

Another potential syntax to consider is using . as the separator, if that's possible.

It would map more naturally to TOML, where the . separator allows you to conveniently define nested tables.

Example:

[dependencies]
namespace-a.package-a = "0.1.0"
namespace-b.package-a = "0.2.0"
namespace-a.package-b = "0.3.0"
namespace-b.package-b = "0.4.0"

...parses to the following toml::Value:

Table(
    {
        "dependencies": Table(
            {
                "namespace-a": Table(
                    {
                        "package-a": String(
                            "0.1.0",
                        ),
                        "package-b": String(
                            "0.3.0",
                        ),
                    },
                ),
                "namespace-b": Table(
                    {
                        "package-a": String(
                            "0.2.0",
                        ),
                        "package-b": String(
                            "0.4.0",
                        ),
                    },
                ),
            },
        ),
    },
}

...which you can also write as:

[dependencies.namespace-a]
package-a = "0.1.0"
package-b = "0.3.0"

[dependencies.namespace-b]
package-a = "0.2.0"
package-b = "0.4.0"
3 Likes

Using the . separator like that would make declaring a dependency on the root of the namespace quite weird.

[dependencies]
serde.version = "1" # is this the crate serde or serde.version
serde.json = "1"
6 Likes

Aah, indeed, it's ambiguous. Oh well.

I think using - as the separator and only applying to newly published crates will lead to a very confusing state in the crates.io world. There are a number of serde-* crates not maintained by the serde devs for example. These crates would now be misleading people as to who their maintainers are, and would make the reservation detection harder than needed, since it couldn’t use - as separator in the code, it would at least need to perform a lookup of legacy crates first.

I’d also point out that most people are only ever going to have one, maybe two active projects that merit namespaces, and the people who would benefit most from this feature are the people who are already publishing multiple prefixed crates today. So this not being available to those existing crates really limits its value, and will make a transition to namespaces painful or slow to adopt.

I think as a user if the namespace solution goes down the crates as namespaces route, it should use its own separator, and at some point there should be a way to allow or convert (depending on how it actually works) an existing crate to be a namespace.

3 Likes

I now agree: after discussing it on the Rust Zulip, it seems adding a new delimiter might not actually be as hard as I thought it would be. See also @withoutboats post above: Pre-RFC: (hyper)minimalist namespaces on crates.io - #13 by withoutboats

Since . is apparently ambiguous everywhere, I now prefer / ala the original packages-as-namespaces proposal. This would also make every existing crate its own namespace, which would be pretty amazing IMO.

3 Likes

It would probably be a good idea to allow old crates such as serde-json to add an alias serde/json if the parent crate serde gives permission. this would prevent needing to publish crates twice for any new releases.

3 Likes