Phantom dependencies

I think it would be useful to have a [phantom-dependencies] table in the cargo manifest. The purpose of this table is to introduce runtime-build version constraints on (e.g. transitive dependency) packages without adding an actual hard dependency edge.

The main (only?) purpose of this is to make achieving a minimal-versions-clean build a localizable concern. Right now, if your library fails to build under -Zminimal-versions, the recourse to fix it ends up being choosing one of

  • file an issue with the upstream library you use that they're not -Zminimal-versions complient, make the argument that they should be and consider noncompliance a bug, and wait for them to publish an update that you can require
  • introduce an explicit dependency edge to the transitive dependency to fix your dependency's insufficient version bound, introducing a fake dependency edge and carrying this around potentially long after it's needed
  • introduce a [patch] and fix it only for local CI builds, and downstream has to also apply the patch if they want to test -Zminimal-versions compliance

[phantom-dependencies] is basically a severely restricted [patch] section, to allow it to be transitively consumed, with the sole purpose of raising transitive dependency version constraints within the semver range but without requiring the package actually be present in the build.

1 Like

Dependencies of dependencies are not a public API and may change anytime. How would you synchronize [phantom-dependencies] across all possible dependency versions?

Do you propose setting a hard lower bound on all possible instances of crate foo which appear in your dependency tree? I suppose that will lead to a lot of failed dependency resolutions, particularly if your crate isn't a binary, so the same crate foo can be depended on by a parallel dependency tree.

In my experience the most common issue with minimal dependencies is that some crate specifies foo = "*", while the earliest versions of foo are broken, horribly incomplete or just a name squat. This is most common with libc, where pre-0.2 versions are written in an ancient dialect of Rust. Perhaps the problem should be solved at the level of those misbehaved ancient versions? E.g. libc < 0.2 could be yanked, or (since I very much dislike yanking) there could be some separate flag which prevents those old versions from being used in normal dependency resolution.

It's likely far from the only issue with minimal-version, but solving it would already go a long way towards making it viable.

It would be the exact same w.r.t. version resolution as a real dependency. That is, within this semver range you get at least that version, but you can still have a duplicate of a separate semver-major version.

This will not and by design cannot select a more recent version than what you'd get normally. It also does not and cannot impact normal maximal-version dependency resolution.

Well, I suppose you could add a constraint which isn't a default caret constraint, but crates-io has been super close to disallowing those for a couple years now, and you shouldn't be using them.

crates-io has forbidden uploading a package with a * dependency since before I started using Rust. I've run into a minimal-versions selecting a * dependency once, and that was trivially fixable by upgrading my dependency to require a nonancient version itself.

In my experience the most common minimal-versions breakage is much simpler: people specifying a dependency version earlier than the one they're developing against (either because it's been a while and this is a new checkout, or they underconstrained it initially out of some principle of e.g. only specifying the semver minor version), and then introducing a use of a newer feature.

Even if you're against yanking, it seems completely reasonable to yank packages that don't build on any 1.0+ version of Rust. These are perhaps the most justified yanks available, since the package doesn't build under any configuration.

For maximal-version dependency resolution, it doesn't matter. Adding a new (caret requirement) phantom dependency will not and cannot by design change the maximal-versions version resolution.

If you want to be thorough, you'd test that your package builds on every version resolution mode that cargo offers. There's no need to check other combinations; if someone gets one, they have a lock file witness that things were working, and if a precise upgrade fails, it's on that package for being minimal-versions incorrect.

The addition of rust-version aware version resolution will complicate things... but if you're minimal-versions correct, then a failure is entirely on the semver compliance of your upstream.

Think of [phantom-dependencies] like a [patch]-lite. It's a hack to work around upstream issues, and once upstream is fixed, you bump to the new upstream version and remove your hack.

Once we have a culture of minimal-versions correctness being important, then [phantom-dependencies] will naturally fall out of usefulness. But minimal-versions correctness won't be widely seen as a think that needs to be fixed until it's available on stable, and it won't be available on stable until it's more usable than it is today, so it's a catch-22 without some hack to allow those who already think it's valuable to patch around a slower-moving upstream.

Well why is libc = "0.1" still unyanked then?

This doesn't matter in practice. E.g. secrets 1.2 uses libc = "^0", which has essentially the same effect as libc = "*".

According to the semver spec, addition of new backwards-compatible APIs requires incrementing the minor version. People are entirely justified in their belief that a patch version difference should contain only bugfixes, and not introduce any potentially incompatible APIs. It's crates like syn which are doing it wrong.

Of course, in practice the line between bugfixes and new APIs is so thin and intertwined, that it is unreasonable to expect people to consistently adhere to the spec on that point.

That's just wishful thinking. My experience tells me that the only proper way to test all supported dependencies is to bruteforce the entire combinatorial space of all possible dependency versions (which is obviously impossible in practice). All kind of things can break and change in incompatible ways between different crate versions.

Also, people with a lockfile don't need any minimal-dependencies, they already have a working configuration. It is exactly when you want to upgrade some of the dependencies when things go downhill.

This makes sense. But maybe it's better to extend the syntax of [patch] to support overriding transitive dependencies? This would avoid proliferating the entities in the manifest, and make it more clear that it's a temporary bugfix.

I'll reiterate that point. A crate may have an entirely different set of dependencies in each of its releases, including patch ones, and it doesn't violate the semver guarantees in any way. Experimental crates may often change their dependencies, and when your transitive dependency count goes in the hundreds, it's almost guaranteed that something will change regularly.

This makes me doubt that crude patching of dependencies for libraries makes any sense and will help in a meaningful way. For binaries, this is more likely to help, since binary crates know the exact versions of all their transitive dependencies at any point in time.

Because the publishers haven't seen any necessary reason to do so. Just because there is sufficient justification does not necessarily make doing so necessary. It also doesn't cover the entire 0.1 line either, anyway.


I honestly thought crates had forbidden = "0" as well...

The point about lockfiles + time leading to the same result is the more important part of the argument, anyway.

Where "upgrade some of the dependencies" includes "adding a new dependency". Scenario:

# downstream
serde = "1.0.100" # lockfile with this version, everything works

# time passes...

# yourlib
serde = "1" # but you use features added after 1.0.100

# time passes ...

# downstream
serde = "1.0.100" # lockfile still has this version
yourlib = "1" # oops, now yourlib is trying to use a too-old version of serde

But as you yourself said, people with a lockfile have a working configuration. The point of testing on minimal-versions is that when they add a dependency on your crate, it also results in increasing the version of the crates you use to the versions you need.

[patch] already works like this. The point of [phantom-dependencies] being separate is that it's inheritable by downstream users of your library, unlike [patch] which isn't, by design.

It does help in one measurable way: minimal-versions correctness. And my underlying point is that minimal-versions correctness isn't only about using minimal version resolution, it's also about normal application of time resulting in having old library versions in use. Yes, crate authors making mistakes happens, but the point is to have some attempt at locally communicating necessary versions.

As to your point, it's impossible to prove that cargo add will never result in a nonworking dependency tree. But that doesn't mean we should ignore iterative steps towards making it better, even if only partially.

The goal of [phantom-dependencies] is to

  • Allow -Zminimal-versions version resolution work a reasonable amount of the time when asking for recent dependencies
  • Thus making it reasonable to stabilize --minimal-versions version resolution
  • Thus making it easier to ask crate authors to be minimal-versions correct
  • Thus removing the need to use [phantom-dependencies].

People can and do introduce direct dependency edges to get the desired effect here. [phantom-dependencies], I hope, is strictly better than status quo.

Binaries already have the mechanism to do so: the lockfile. Libraries have no mechanism to even try.

1 Like

This facility would help out with certain platform-dependency situations, which have nothing to do with minimal-versions. Some crates refuse to build or run properly on certain platforms without certain features enabled; e.g. getrandom and instant won't run on wasm32-unknown-unknown by default, and winit without default features won't build on Linux. So, my crate that's compiled to a wasm module needs to list those dependencies (two versions of getrandom`) in order to enable the features, even though it has no direct usage.

Regarding “Dependencies of dependencies are not a public API”: for the situation I'm looking at, lock files already provide “use these versions until I say otherwise”, which mostly freezes the dependency graph — but there is no way (and probably shouldn't be) to say in a lockfile “enable this feature, even though no individual crate says it needs it”. So, I think [phantom-dependencies] would improve this situation, by allowing these oddball dependencies to be declared as what they are.

1 Like

That's a great use case!

I think the name [phantom-dependencies] communicates the purpose well: these are packages which you do not actually directly depend on, but which need to be present in the dependency resolution for someodd reason.

But which you need to further constrain, if they appear in the dependency resolution. My reading of the intent was that when you use an updated version of your dependency which uses a (semver-incompatible) updated version of its dependency which you have a phantom-dependency on, that phantom-dependency does not force both versions to be in the build. (Which will be really fun to resolve if there’s any non-caret constraints around, but I think should be possible).

This seems like an issue for the feature usecase, you cannot predict when the dependency will upgrade its dependencies, so you’re not forward-compatible with enabling the feature on future versions of the phantom-dependency.