Feedback on `cargo-upgrade` to prepare it for merging

tools and infrastructure cargo
41

I'll have to look more at your proposal but my concern with these fields is I suspect most people do not care about bumping the absolute position but care about bumping the relative position. I want to upgrade across incompatible releases, regardless of whether the field is major or minor.

We could make --major be relative but I would be concerned with the 0.x to 1.y jump and with user confusion who would expect it to be absolute. With the 0.x to 1.y jump, I'm also putting on my cargo-release hat which has to have support for absolute positioning to make the 0.x to 1.0 jump and I'd rather not have the confusion over --major having different semantics depending on what tool is being used. While cargo-release isn't on the path to integration, I think it and tools like it are still visible enough and important enough to consider.

1 Like
42

This is also a bit of a terminology thing, since cargo extends the semver spec (in a defacto common way).

There's two ways of describing the extension. In my personally preferred way, it's the "0.MAJOR.MINOR extension," where the second position is the major version number while the first, the actual major version, is still 0. (This also allows distinguishing between 0.MAJOR.MINOR and 0.MAJOR.PATCH conventions; whether you're bumping the second position just for breaking changes or for all feature updates, respectively.)

More correct, though, is to say that MAJOR is still 0, and that it's the "0.MINOR compatibility extension". The semver version 2 spec says that all 0.* versions are to be considered arbitrarily incompatible. The extension is to say that all 0.N.* versions are considered compatible (unless N=0). Importantly, this avoids conflating the real MAJOR version of 0 with the nonzero version fields.

The former is perhaps a concession to the use of zer0ver, and the latter more strictly semver.

I do think cargo should strive to stick to semver-accurate naming (and thus the latter isomorphic-up-to-naming extension here). Thus update --major should refer to the semver MAJOR version number, not the leading nonzero version number (which may be the MINOR or the PATCH version).

1 Like
43

I suspect there is a large enough subset of users that would appreciate a --patch flag, whatever it gets called

Most cargo commands avoid positional arguments

  • To allow users to see an option's values by making the value optional (cargo update -p)
  • To avoid ambiguity in parsing the above (which is unlikely to be a problem in practice as its most likely to be the trailing arg)

As for csv@latest, did you have in mind that it would be the same as csv or would it allow upgrading across incompatible versions?

And it would be possible to specify a lower version to downgrade as well as upgrade with the same syntax.

I do like the idea of allowing downgrades. Maybe that would be a better way to describe the --incompatible flag is "force to work regardless of current version req". Now, what to summarize that for a flag, I'm unsure

  • --incompatible could refer to the version req rather than semver
  • --force but there are too many different things you could be forcing and it would still fail if you don't include --save (at least in the current plan)
  • --ignore-req but I'd like us to preserve precision, so we wouldn't be completely ignoring it

Unless someone has a better name, I think I prefer --incompatible.

I do think --compatible and --incompatible or incompatible could be nice, but I can't imagine myself wanting to type them interactively if I could avoid it

Because of length, chance for typos, or something else?

1 Like
44

I'm just going to echo one more time: --patch --force which upgrades version specification precision?

= "(\d+\.\d+)"/= "$1.0"/g functions, and restricting it to **/Cargo.toml may even be near[1] free from false hits, but I really do hold it as axiomatic that there should be a way to upgrade --to-lockfile such that cargo upgrade --to-lockfile --magic && rm Cargo.lock && cargo generate-lockfile --resolver=minimal-direct gives you the exact same direct dependency lockfile versions, and that this should function without downgrading the currently used versions, no matter what the current Cargo.toml state is in.

Cargo should be imho be opinionated that caret dependencies should specify the patch version. Not bumping to the most latest patch version needlessly is preferable for ecosystem compatibility, but not specifying the current patch version when adding a dependency is a misguided approach to increasing compatibility. (And IIRC cargo add always specifies the patch version and doesn't allow requesting lower precision without just directly specifying the used versionreq.)

Personally, I'd be fine with cargo upgrade always upgrading precision (iff the version is actually upgraded), and answering requests to preserve precision on upgrades with "no, that's an antipattern to underspecify caret version requirements," but I understand that I'm likely in the minority here.

  1. It could have a false hit in the metadata table, but I don't think elsewhere in the real Cargo metadata. ↩︎

1 Like
45

To clarify, I took --patch to mean "Restrict available packages consider to those with different patch numbers". I saw this being useful regardless of any --save flag. cargo update --patch would not update to new minor versions and cargo update --patch --save would then update the version requirements accordingly if they include enough precision.

As for cargo add vs cargo upgrades handling of precision, see my earlier post

46

I'm still stubbornly using the deprecated --all flag, simply because --workspace is too long to type. Can you keep --all or add a shortcut to all cargo commands for --workspace (e.g. -W).

47

I've tried the latest version of cargo-edit, and I can't find how to actually make it upgrade dependencies to the latest patch version available in the crates.io registry.

If I just do cargo upgrade --all package_name_here, it finds that the latest version in the registry is "compatible", and does not change Cargo.toml.

I've tried cargo upgrade --all --to-lockfile package_name_here, but I happened to have less-than-latest version in the lockfile, so it ends up upgrading Cargo.toml to a not-latest version.

So it seems that it requires cargo update; cargo upgrade --to-lockfile combo to actually require latest patch version. I don't like that, because I have git deps in my project, so cargo update is slow for me, and --to-lockfile --workspace is verbose. I'm used to cargo upgrade --all.

1 Like
48

Yes, cargo-edit master has it split between workflows. The proposal listed in the top post changes this. The main challenge is how to express it so library authors can upgrade through breaking releases while no changing anything else, if they desire.

49

I'm posting to join other people confused by the latest changes.

mqudsi commented on August 18th 2022:

I'm also very confused by the latest changes. IIRC, running a bare cargo upgrade used to mean "upgrade even if it's not compatible and let me sort out the issues" but now it doesn't do anything at all (in cases).

My use case is the same. I have a workspace containing about twenty projects, each with their own dependencies. I want to run cargo upgrade to update all the requirements to the very latest version in every Cargo.toml (ignoring compatibility). After this I can look at the Git diff / run my unit test / check if everything is OK. This helps me use the most recent version of all my dependencies without having to check their crates.io page manually.

With the latest cargo edit version I get a message with new compatible version but it does not update anything. Is there a way to get the previous behavior back or is not available at all at the moment?

1 Like
50

At the moment, to get exactly the old behavior requires doing cargo update && cargo upgrade && cargo upgrade --to-lockfile.

Thank you for sharing your experience and your concerns.

  • Could you speak to why you are wanting to update all of your version requirements?
  • Could you speak to how the proposed behavior under Known Points of Concern 1 would work for you or not? Having your input on expected future behavior would be useful so both of us are not surprised on the next iteration of behavior.

My expectation is that we will make cargo upgrade try to match the proposed behavior and publish that before doing the merge. I have not started on that re-work of cargo-ugprade yet as my focus is elsewhere at the moment (clap v4).

51

Not speaking for @demurgos, but I'd like this as well: I want to upgrade the incompatible ones so that I can use the latest major version, and I want to upgrade all of them to the latest versions because that's what I'm testing with and can confirm works. I'd like that to be simple functionality to reach from cargo update, since it was trivial with cargo upgrade. I'm fine with it becoming an option to update, but it shouldn't require multiple options to update (and definitely not exclusively long options).

The behavior described in that section seems like it would require passing more options for the common cases. --save is the common case for upgrade, and while it's non-default behavior for update, I think it should be implied by options that don't make sense without it. (I wouldn't object if this checks if the manifest is version-controlled first, though.)

So, working it out incrementally:

  • cargo update with no options needs to keep doing what it currently does: update the lockfile to the latest compatible versions. It could, however, check for updates to incompatible crates as well, mention them, and suggest the options to use for additional behavior.
  • Upgrading incompatible dependencies thus needs to be a non-default option. -i/--incompatible seems reasonable. However, I don't think it makes sense to either upgrade the lockfile without the manifest or error out; I think this should just imply -s/--save, rather than erroring out. (As above, this could error out if Cargo.toml is not tracked in version control though.)
    • I do think it also makes sense to have the behavior of cargo upgrade --skip-compatible available, though it can be harder to reach. cargo update --only-incompatible perhaps and defer the question of a short option? And again, that should imply -s/--save, because it doesn't make sense without -s/--save.
  • -s/--save is useful even without either of those options, to upgrade only compatible dependencies. (Bikeshedding: this could also be spelled -m/--manifest to make it clear where it gets saved, or something else more explicit.)

Net result:

  • cargo update doesn't change
  • cargo update -i does exactly what cargo upgrade used to do
  • cargo update -s is new and useful
  • cargo update --only-incompatible does what cargo upgrade --skip-compatible used to do
5 Likes
52

Background

I manage a medium sized platform for developers creating web games. This is a non-profit community project with about 20 games only supported by donations and voluntary help from contributors. We don't have customers or deadlines. This means that a little bit of instability is not a deal breaker for us: we can spend some time fixing breaking changes introduced by dependencies to get a better project afterwards.

Why I want to update all version requirements

Could you speak to why you are wanting to update all of your version requirements?

The two main reasons is that I believe newer versions are better and that regular updates are easier to manage. Letting dependents use older versions of transitive dependencies is a source of issues.

Newer is better

New versions often have new features, fixes, performance, documentation improvements. I want my projects and dependent projects to benefit from these improvements as soon as possible. This also aligns with cargo's default range for resolution (^, highest version with matching first non-zero value). I use the default range for my projects.

What happens is the following:

  1. My dependents install my lib, and most of them will get the latest compatible version of the transitive dependencies.
  2. So I should test my lib with the latest compatible version of my dependencies.
  3. So all my dependents should use the latest compatible version: it's the one that was tested.

If my lib foo:1.0.0 depends on bar:^1.2.3 and they release bar:1.2.4 then I should test foo with this version and update my requirements to bar:^1.2.4.

What can go wrong with a mismatch between the tested version and older requirement

I can't just leave ^1.2.3 when I release foo:1.0.1. My dependent projects will already have a Cargo.lock with bar:1.2.3. Updating foo from 1.0.0 to 1.0.1 would keep bar:1.2.3 if I don't change the requirements. This is because cargo is conservative when a Cargo.lock already exists. I need to update my requirements to force an update of the transitive dependencies.

This is not some theoretical concern: I had this exact issue with scrypt because it was tested with a more recent patch version but did not update its requirements properly.

Large windows of compatible versions are riskier

The more general feeling I have is that semver ranges are good as they allow to get the benefits mentioned at the start of this section for all transitive dependencies, however maintainers should also strive to keep the highest requirements possible so the number of possible versions remains fairly low.

This allows to reduce variations and accidental compatibility problems. From this point of view, cargo upgrade is the complement of cargo publish: they both move the window of compatible versions, just different ends.

In particular, I want the latest version even if it is a semver breaking change. I am working on my project at the moment: let's update to the latest version and push it to dependents, as it will be the only supported version going forward anyway.

So count me in the opposition to low requirements :smile:

There seemed to be general interest in keeping library version requirements low within a major version but to upgrade across major versions.

Regular updates are easier

I described why I want to use the latest version and pass them down in my requirements. Being able to track the latest version becomes important then. I occasionally go through all my Cargo.toml files and check crates.io if there are new versions. However doing this manually is a slow process.

An important use case for me is then to use cargo upgrade as some sort of local "dependabot". I can run it in my project and it will update all my manifests. (Small aside: I wish cargo upgrade had better support workspaces, right now it bails out as soon as it hits a dependency not on crates.io even if it exists in the workspace).

This usage of cargo upgrade lets me check my Git diff, try to compile, run my tests. In the majority of cases, this allows me to confidently commit the updated dependencies.

For situations when there are breaking changes, this usually does not involve a lot of work due to to Rust's help, changelogs (or simply checking the commits). However, it becomes harder if I don´t update regularly as then more changes will have piled up. Having a fast way to upgrade all dependencies is important to me to avoid maintenance spikes.

By the way, I much prefer a command to update all the manifests and then let me edit the changes in my editor instead of an interactive command. It's faster for me to use my Git diffs and editor if I need to undo a couple upgrades. This is my main issue with yarn up: it's cumbersome to update everything, the default for this is to use the interactive mode (you may add yarn up to the prior art).

Opinion on the proposal / points of concern

First of all, sorry for my previous message which was mostly originating from the GitHub issue on the latest cargo-edit changes and was a bit out of topic as it did not reply to the proposal at the top of this thread. I am a regular visitor of this forum, but post fairly rarely, As I was asked, I'll share my opinion on the proposal

Could you speak to how the proposed behavior under Known Points of Concern 1 would work for you or not? Having your input on expected future behavior would be useful so both of us are not surprised on the next iteration of behavior.

First of all I really like the overall feature and proposed notes: keep the existing precision or skipping explicitly pinned versions.

My main disagreement is with the following point:

  • By default, it deals with breaking changes only. To instead upgrade minimum versions, pass --to-lockfile.

As I explained above I believe that we should strive to have the highest requirements possible to limit variance and ensure dependents actually get transitive dependency updates. I feel that updating all compatible requirements by default, and having a flag for incompatible requirements may be a better default.

I also find the flag --to-lockfile confusing, in my mind it should be --from-lockfile as we apply versions from the lockfile into the manifests. I guess the intended meaning is "to [version in the] lockfile", but it's not that clear to me.

Regarding the concern 1 ("Distinction between cargo update and cargo upgrade may be confusing"), I really like the approach to merge update and upgrade and the proposal currently in the original post (initially posted there). In particular, I like the formulation " upgrade interacts with workspace members (manifests) while update interacts with the workspace itself (lock file)". Having --save as a flag to pick the mode works well for me.

Anything bringing the manifest and lockfile closer to each other is a good thing for me. So having a single command to manage both kinds of files is pretty clear.

My use case should be well covered by the proposed cargo update --incompatible --save flags.

I like this idea a lot, --save is a bit ambiguous; --manifest or similar is clearer to indicate the new mode.

5 Likes
53

I'm a bit hesitant about having a --incompatible imply --save

  • Unless you carefully read the docs, it would be unclear what user experience is expected
  • While with a fresh start I would prefer a --dry-run, some people have brought up dry run being the default with an explicit flag to modify

imo implying --save to me makes me feel like this direction is faulty and that we should either have a separate command for the manifest or to keep them combined but redo the UI by deprecating the old command and creating a new one.

54

Before I get to anything else, please open an Issue with reproduction steps

55

To me, --manfiest reads as an option that takes a value.

To help spur other thoughts, going to rephrase this

  • We save or write to the manifest path
  • We modify or overwrite the version requirements in the manifest path
  • We write-through updates in the lock file to the manifest path
56

Thanks for writing out your thoughts on this.

We've taken the approach "newer is better" with cargo add because we cannot assume what features people might use after they add, so full precision latest is the safest option.

I will say that just because cargo/rust today doesn't help with testing minimum version ranges, I don't think we should abandon keeping them low. There is an unstable feature to use minimum versions for requirements. The main issue with it is that it requires whole-ecosystem buy-in. A variant has been proposed where only minimums of direct dependencies are selected so you can adopt it on your own. Another way of improving things is if we had a clippy lint when using a function newer than the minimum compatible version. This could be build on top of stablizing the #[stable] attribute if others were to use it.

In talking of the benefits of regular updates, we should also look at the downsides

  • Prevents downgrades to workaround bugs
  • If always taking the absolute latest (without taking into account MSRV) then you become more dependent on others for your MSRV (e.g. clap today has a lower MSRV than one of its dependencies)
  • If you are doing code or license audits, being forced to upgrade all dependencies when upgrading one can be a lot more expensive
  • More frequent long compiles as more needs to rebuild

imo the policy of when to upgrade dependencies should be left to the final binary barring needed features or bug fixes.

However, it becomes harder if I don´t update regularly as then more changes will have piled up. Having a fast way to upgrade all dependencies is important to me to avoid maintenance spikes.

Could you expand on the maintenance spike within a major release? The main case I can think of is a crate like clap which does deprecations throughout its development. Right now, that approach is rare though I would like to see more of it as I feel it makes upgrading across major versions easier.

57

I don't think we should attempt to push that on users of a new tool or new functionality of an existing tool, if those users are not passing such an option. Minimal versions is something you can opt into, with a great deal of challenge as you'll be debugging other people's dependency versions.

58

I think the last of those feels excessively complex. save seems preferable of all of those options.

59

FWIW, this is what -Z minimal-versions testing would catch.

Note that this just puts work up-front when updating things for your dependees. If I need to update your dependency for $reasons, dumping "oh, please also update for changes in 6 of my deps that we share that I'm forcing because I follow HEAD" on me at the same time makes it easier to say "uhh, maybe I need something simpler".

Especially when you're doing things like this.

60

If we were modifying a file that wasn't tracked in version control, I would agree with that: dry run by default, flag to modify. However, when modifying a file that is tracked in version control, I feel like the best user experience is to just make the change, and then the user can run git diff and decide whether and how much they want to commit.

While I can imagine trying to teach users that "update is for the lockfile, upgrade is for the manifest", that seems to me like a more complex distinction than those words suggest.

I would propose that cargo update is the relatively low-friction low-impact default: it modifies the lockfile but doesn't touch the manifest, and it can print additional information about additional changes it could make.