[Pre-RFC] [Cargo] Support vendored/local-directory alternate registries

internals
1

[requested to file a RFC in https://github.com/rust-lang/cargo/pull/7188, so opening a pre-RFC thread here first]

Cargo supports alternate registries, which allow for packages from multiple sources (under independent namespaces) to be included in the same project. Currently, these work by specifying the URL of a git repository, which includes a configuration file naming a download server. This works well on many desktop use cases, but when integrating with existing build systems things can get more complex - in particular, they may have their own ideas about build reproducibility which do not allow access to external servers, which control which source or crate versions are visible to a build, and have their own version management rules which do not play well with Cargo.lock files.

With crates.io, these problems can be dealt with by using source replacement; a pre-build script can examine the Cargo.toml and/or Cargo.lock files, resolve them based on a frozen snapshot of crates.io, and download them via the external build orchestration framework. However, it does not appear possible to do this with alternate registries - as far as I can tell, they require a HTTP server for the actual download of the crates, at a minimum, and do not support source replacement entries.

While it would be possible to implement source replacement rules for alternate registries, this is a bit strange - we'd have to specify a bogus URL in the [registries] section, only to replace it with a local-directory source replacement rule. Therefore, I propose simply allowing the alternate registry to be directly specified using a local-directory statement in the relevant [registries] section.

1 Like
2

I'm having a little trouble understanding your use case. A concrete example would be helpful.

They support file:// urls to download locally.

A path can be specified like this:

[registries]
my-registry = { index = "file:///path/to/index" }

There are also other methods, like local registry sources that may be applicable.

3

Aha! I didn't realize file:// URLs were supported (I might send a doc PR in that case instead). The tree layout is a bit strange for a filesystem tree, and it's inconvenient to have to construct a git repository here, but it'll work to unblock us.

It's hard to provide a concrete example, since this is all about integration with an internal build system, but essentially we want to build an alt-registry of our first-party crates, and we can't reach out to an external server. Today, we merge these into a vendored snapshot of the crates.io tree, but this has a number of unfortunate side effects (e.g. cargo outdated doesn't work properly as our tooling subsets the crates.io tree to just the versions we're using).

As for local registry sources, it seems that they currently only work for crates-io - the source replacement infrastructure is hardcoded to only be queried for crates.io (with the exception of some handling on cargo install), and the alt-registry config parsing logic today doesn't understand the local-directory key in a registries entry (this is what this pre-RFC proposes to add support for).

4

The directory registry mechanism should work as a replacement for any registry. If it doesn't, that's a bug and should be fixed.

5

If we treat this as a bug, do we expect that the registry name should be used to locate the relevant entry in the sources map, or should the registry index URL be used? And if these are inconsistent, what happens?

e.g.

[registries]
[registries.foo]
index = "http://this.is.a.bogus.url"

[sources]
[sources.foo]
registry = "http://some.other.url"
replace-with = "foo-replacement"

[sources.foo-replacement]
local-registry = "/foo"

[sources.bar]
registry = "http://this.is.a.bogus.url"
replace-with = "bar-replacement"

[sources.bar-replacement]
local-registry = "/bar"

Do we expect the alternate registry foo to be mapped to the local-directory /foo or /bar?

Edit: I'll also point out it's a bit awkward to explicitly write an external URL in the same configuration file where you override that external URL to point somewhere else.

6

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.