[Pre-RFC] [idea] Cratespaces (crates as namespace, take 2... or 3?)

This is a slightly different take on the crates and namespaces suggestions made previously. Previous RFC's and discussions are linked at the end.

Pre-RFC: Crate-spaces (crates as namespaces)

Motivation: There's been a discussion around mono-crates versus many small crates for project like tokio.

A group of small crates provides a number of benifits - granular compilation, independent versioning allowing for more flexible iteration, and an easy way to control how much code is compiled, compared to what is used.

A mono crate can control compilation through features, is easier to audit from an author perspective (bigger crates == fewer authors) and is easy to ensure all the bits work together

This proposal seeks to combine the best of both worlds.

Purpose: Allow for grouping crates into a cohesive whole, allowing for fine grained compilation while preserving crate metadata transparency, for ease of auditing and discovery.

Considerations:

  • Backwards compatibility is pretty important - we don't want to arbitrarily cut off older software as
  • Not adding another sigle in code for using namespaced crates

Summary

Each top-level crate may act as a namespace. This doesn't prevent the crate from being used as a standalone crate. It might even be empty.

Additional crates (subcrates) may be published as children crates by the author(s) of the crate-space. These children crates are not directly accessible as top level crates (and therefore unaccessible to older rust versions). They may however be republished outside the cratespace by the author for backwards compatibility. There is no difference between using a child crate through the cratespace or as a standalone crate for versioning/interopability purposes. An external crate may be added to a single cratespace as a child crate, if the cratespace author and crate author agree. The primary use-case is for building a cratespace from existing crates.

A cratespace may import and publish an external crate as its own subcrate, with the caveat that is it marked as a voucher crate, not a child crate.

Subcrates are referenced as submodules of the main crate; they may be accessed via the path crate::subcrate in code. Subcrates may be versioned independently. A cratespace may publish a versioned version manifest, which assigns versions for all of its subcrates. Subcrates may be imported independent of the core crate space, but will still need to be referred to through the top.

Examples

Serde could act a cratespace. Then serde-derive could be kept, but republished under serde as derive and accessed as serde::derive in code. The serde_rustler crate could be added as a voucher crate to the serde namespace as rustler and then referenced in code as serde::rustler, if so desired.

Downsides

Maintaining backwards compatibility requires extra work - it's easy to be part of a cratespace or be backwards compatible, but being both requires effort.

In addition, this implementation generally adds complexity, and has a number of edges with undefined behavior. And despite care to not encourage things like user123/http, it is still possible. Multiple versioning schemes (independent vs manifest) are both desirable, but having both will probably lead to confusion.

Alternatives and/or Possible Tweaks

  • Domains as namespaces
  • Don't implement namespaces
  • Other namespace schemes
  • For importing, use _ instead of :: and instead use crate_subscrate
  • For importing, drop the cratespace:: prefix
  • Require child crates to be published as separate crates first (to enforce backwards compatibility)
  • Publish a crate with internal only dependencies (e.g. the dependencies aren't visible to the world); added per @CAD97

Other Considerations

  • Can we use tricks here to help with coherence (with voucher crates?) - how would that affect dependency resolution to avoid having two crates with the same impls.
  • What happens when a subcrate is added - who has write/author permission for that crate?
  • In this setup, it would be nice to allow a request for voucher crate status mechanism. It would probabl be safest for voucher status to only apply to the set of crate versions published by the approved authors, and new authors would require a new approval... right?
  • Generally, voucher crates will need some serious consideration to avoid security issues, I think.

Previous Discussions and RFC's on Namespacing:

1 Like

Another related but distinct potential feature is internal-only crates: publish a crate with crate dependencies that aren't visible to the outside world at all. Then the crate split becomes merely an implementation detail.

4 Likes

How does that work on the network level? When you download cratespace.crate, does that tarball include source code for all of its subcrates, or does cargo make separate requests for subcrate.crates?

Are cratespaces the same thing as Cargo workspaces? How does that grouping work locally before publishing?

I haven't thought too hard about these points, so hopefully my answers make sense :slight_smile:

I'd be fine with it making separate requests for each.

I envisioned it working as the cratespace root crate listing what subcrates it contains and each subcrate listing which cratespace it belongs to, in order to support crates that aren't in the same workspace, which means the cratespace root and its subcrates would be published separately. And when only one of those is done, the status of the subcrate would be "pending" so to speak, in the crates.io UI. You could see what the person was wanting to happen, but it has to be approved through both.

That said, I think it would be nice to simplify it for workspaces. It'd be nice to add a flag to workspace to make it into a cratespace, without any extra work necessary, and then publishing tries to publish any crates that have changes.

Does this idea have a thread/rfc/github issue etc?