I'm curious to hear your thoughts on this. I've actually been meaning to propose the opposite: we should always check-in lockfiles, not just for binaries.
Hey, I'm also interested in your argument against committing lockfiles.
My opinion is that all projects should always commit their lockfiles. The reason is reproducibility.
crates.io).cargo update).In my opinion the benefits of long-term reproducible builds far outweigh the need to be explicit when you want to check behavior with the latest compatible dependencies. In particular they are invaluable when investigating regressions or trouble-shooting build errors. If a crate fails to build locally, but I see a commit with a lockfile and green CI it narrows down the source of errors to factors outside of the control of cargo.
To be honest I'm still a bit baffled by this recommendation in the cargo book:
Why do binaries have
Cargo.lockin version control, but not libraries?[...]
For libraries the situation is somewhat different. A library is not only used by the library developers, but also any downstream consumers of the library. Users dependent on the library will not inspect the library’s
Cargo.lock(even if it exists). This is precisely because a library should not be deterministically recompiled for all users of the library.
A Cargo.lock in a lib repo is obviously not for users but for lib devs. A library still has binaries, they're just called through cargo test instead of cargo run; why should they be non-deterministic by default? A Cargo.lock does not prevent running your tests against the latest dependency versions, it's easy. What's hard is manually resolving the dependencies at a given time when you forgot to commit the lock file in the first place.
This is also the stance of Yarn, which is IMO the most reliable package manager for Node:
Which files should be gitignored?
[...]
yarn.lockshould always be stored within your repository (even if you develop a library).
Yarn actually goes even further and recommends storing the Yarn version itself in the project. I recommend also reading their arguments (they have other ones, such as a missing lock file not being sufficient to ensure you use the freshest dependencies anyway). cargo can piggyback off rust-version so we probably don't need a cargo-version.
Also to have a message on the topic of this thread. A few months ago I posted a comment about a previous version of cargo upgrade complaining that it was harder to pull the latest version to test against them. Since then, cargo-edit was updated a few times and I'm very happy with the current iteration of cargo upgrade. Thank you for your work.
I'm a huge fan of this even though it seems relatively unpopular. Bisection is often difficult/impossible without a Cargo.lock. Even getting a clean build on a stale repo can be problematic.
This has lead to requests for a Cargo time machine that could try to do the resolution based on a historical index state, but there's a lot simpler solution: just check in Cargo.lock.
This is the approach Go's modules take, and I think lockfiles for libraries are maximally powerful combined with Go's notion of Minimum Version Selection (research!rsc: Minimal Version Selection (Go & Versioning, Part 4)). Obviously that's a much bigger change than simply including them.
I'm very curious to hear about this as well.
cargo-edit 0.12.0 is out. I have not (yet) revamped the UI but did smaller steps
--recursive changed from being on by default to the default being based on if compatible upgrades are being doneCommitting a lockfile doesn't mean it must be used. People are always free to rm it, do cargo clean, and build from scratch.
The lockfile is more like a documentation: it last worked with this particular set of dependencies. Useful for debugging all sort of problems with dependencies, like regressions and semver violations.
Looking to summarize a recent discussion on where to go from here. I've tried to go back and include some details from the thread but I likely missed some.
Expected user operations across cargo update and cargo upgrade
tokio_03)Note: "compatible" and "incompatible" are relative to the version requirements and not semver (though thats the default version requirement operator)
cargo update today
Cargo.lock, not Cargo.toml
cargo update -p foo@ver, @ver is used to disambiguate foo within the dependency tree (must be full version)
--precise ver takes a full version for replacing ver in the lockfile and must remain compatible--aggresive to recursively update foo
Considerations:
[[bin]] maintainers may want bulk updates of everything, most likely with a focus on compatible as incompatible might need hand updates[lib] normal/build deps: maintainers may want compatible updates by hand. incompatible bulk updates are good for checking of it works but sometimes they will need to be done by hand[lib] dev-only deps: like [[bin]]
-Zdirect-minimal-versions
-Zdirect-minimal-versions or a way to update requirements without updating the lockfileProposal 1: Deprecate cargo update in favor of cargo upgrade
cargo upgrade (formerly cargo update && cargo upgrade --to-lockfile)cargo upgrade --incompatible / cargo upgrade -i
cargo upgrade -p foo
-i, --pinned
cargo upgrade -p foo@verreq
--pinned needed?cargo update -p foo --precise ver?Proposal 2: Separate Commands
cargo upgrade only does incompatible (formerly cargo upgrade --incompatible allow --compatible ignore)cargo update --save does compatibleProposal 3: Merge Upgrade into Update
cargo update stays the same except...cargo update --save (formerly cargo upgrade --incompatible false --compatible true)
--save is needed feels off to mecargo update --incompatible (formerly cargo upgrade --incompatible true --compatible false)cargo update -p tokio_03 de-sugars the dep name to package name + version (tokio@0.3.12)cargo update -p foo --precise ver
--precise and not allow controlling the precision or operators, do we hack up --precises behavior, or find a new flag?ver is incompatible, do your need --save, -i, or either?cargo update --precise --save to fully specify all version requirementscargo update && cargo update --save --locked would be an error, mirroring cargo adds behavior which has --locked apply to the manifest as wellNote:
--edges flag which works when talking about graphs but not really in this contextlinks?
git2
I think the next step is updating cargo-edit to one of the proposals for us to gain first-hand experience with how it works out
I still don't think we should have --compatible true and --incompatible true; let's just spell those --compatible/-c and --incompatible/-i. If you pass either one, you just get that one; if you want both, pass both.
Of the various reasons given to not merge upgrade into update, the main one seems like the inconsistency of having to give --save to upgrade the manifest for compatible upgrades but not for incompatible upgrades.
I think that's worth not having two commands or a deprecated command, though, because I think it's a natural consequence:
--save would be redundant there.-c/--compatible option, and then having cargo update -c and cargo update -i both save changes to the manifest.So:
cargo update stays the samecargo update -c upgrades compatible and saves to the manifestcargo update -i upgrades incompatible and saves to the manifestcargo upgrade -c -i upgrades both compatible and incompatible and saves to the manifest.The options I gave dropped --compatible <value> / --incompatible <value>. The difference is in the details of what happens if no flags are passed in and whether both flags exist.
I disagree when it is as destructively described as --save.
However I think this is subject to how the flag is framed and framing it a different way (your --compatible) bypasses that. Sorry, I had forgotten to call this out when doing my write up.
I'm still a little cautious of this approach as there is still a gap in the concepts of --compatible and --incompatible . This isn't a "no", just a "eh".
Then I misunderstood your parentheticals; I thought you were giving "abbreviated form (long form)" rather than "new form (old form)".
Thanks for pointing out that unclear point; I've tried to clarify it
I kinda disagree here. For bins, if you're doing an update --incompatible at all, the goal is most likely to get all dependencies on their latest version, and there's no reason to not do so for dependencies that haven't released a breaking bump.
If you're being more precise about it, then you'd (either do so originally or back up and) probably do each incompatible dependency upgrade in an independent step and commit, not attempt to do all of them at the same time.
I still don't really see --incompatible true --compatible false ever really being a desirable workflow[1]; either you want to update all dependencies in one step, or you want to update single (trees of[2]) dependencies at a time to minimize the incremental patches needed to resolve the upgrade.
So my counteroffer would be I guess roughly
update writes to lockfile (as today)update --save writes to manifestupdate --breaking is an error, suggesting one of the belowupdate --breaking --save writes to manifest, upgrades allowed to cross compatibility rangesupdate --breaking --precise implies --save for just the specified dependencies (any transitive updates are lockfile only and not saved unless --save is also specified)A interestingly meaningful command might then be cargo update -bsp tracing-subscriber; this would upgrade your manifest dependencies on tracing-subscriber and tracing to the most up to date potentially breaking release.
Interesting (but potentially poor) idea: differentiate between version = "1.0.0" and version = "^1.0.0"; --breaking will update "default bound" version requirements across compatibility ranges but not move requirements using an operator across compatibility ranges. Rationale: using a non-default requirement mode is an advanced use case (which required manual editing of the manifest at some point rather than the use of cargo add and upgrade) and the user probably knows what they're doing and want better than the tool can guess it. As an escape hatch, a --force flag will change back to the default bound requirement again (if an update outside of the bound exists) and do the update.
That idea is why I used --breaking instead of --incompatible; I'm focusing on the fact that an incompatible upgrade of the default requirement mode is potentially breaking (and thus not done without an opt-in), rather than that it's incompatible with the version requirement.
Even for libs that want to keep low requirements within compatibility ranges, I'd describe what they want not as "upgrade incompatible to the most recent" but "upgrade all to the lowest compatible in their most recent compatibility range;" it's not a difference in the upgrade they're doing but in the version selection preference. ↩︎
Things are a bit more interesting when multiple dependencies related by a public/peer dependency are involved, as you do want/need to upgrade them together. ↩︎
With the current cargo-edit version is there any way to upgrade all dependencies for one crate within a workspace? Or even upgrade a version requirement in one crate but not another when multiple in the workspace have the same dependency? It seems like it's only capable of operating in --workspace mode currently. (I have a few repositories containing separate library and binary crates so I commonly want to treat their dependencies differently).
Currently, everything is workspace-wide. The motivations were
I think this is the first time since that change that someone has expressed interest in handling upgrades per-workspace-member. Feel free to come up with a counter proposal for how it can work. I'm assuming supporting this would push us away from merging cargo-upgrade and cargo-update.
I like --breaking over --incompatible. FWIW, I don't like update --save because IMO update without --save also saves (to the lockfile). I would maybe go over something like --requirements instead to make the difference more obvious.
I disagree. I never like bumping semver-compatible dependency requirements to force them to their latest version, because it's relatively common that the latest version of something will break/be yanked, which is easier to repair if it's just the lockfile.
Also in terms of precision, I usually don't actually need the newer dependency for a compatible update, so it also goes against the grain of maintaining minimal(ish) versions.
@djc Honestly, if we can agree on a structure of:
cargo update - same as beforecargo update --something (plus short option) - upgrade (just) compatible versions and write to manifestcargo update --something-else (plus short option) - upgrade (just) incompatible versions and write to manifestcargo update --something --something-else (plus short options) - upgrade all versions and write to manifestthen I really, really don't care much what color we paint those two bikesheds. People can learn whatever the option is as long as it's even slightly intuitive.
Yes, I think that design makes sense.
cargo update --save never really sat right with me. To help work through this, @josh and I had some further conversations on this.
Based on where this landed, I would ask that if people propose alternative solutions, to frame them in terms of what role the related commands are meant to fill and how the solution fits within that role.
For reference, our shared priorities are
cargo updateCargo.lock and Cargo.tomlOur primary use cases we are targeting are:
cargo update)Our secondary use cases are:
Some open questions we had
tokio_03 is pinned or not?
--exclude these dependencies but that might be a bit of a pain to always remember to doCurrently, cargo update is focused solely on Cargo.lock editing
name@versionVersion requirement editing is different in that
And as a reminder of the CLI:
cargo update -p foo # select a non-ambiguous `foo` and update it
cargo update -p foo@ver # selects a specific `foo` to update
cargo update -p foo --aggressive # also update dependencies
cargo update -p foo --precise <ver> # select a specific version
cargo update --locked # fail if the lockfile will change
Note: cargo add --locked will also fail if the manifest will change
Some design tools we can consider include
cargo <cmd> --help and cargo --list will point people to the new namefoo@ver syntax for versions and version requirementsCargo.lock mostly align with Cargo.toml, making it easier to conflate the two commands (whether merging them or keeping separate but de-emphasizing update)Through this, I realized that the core of my concern with our previous attempts at a single command is that it feels like we are shoehorning new behaviors into cargo update rather than making the behavior cohesive.
cargo update --incompatible on a blog post, can I predict what will happen if you do cargo update? No, because --save is needed to match behavior--package be both for package IDs and dependency names to make some of the cargo upgrade workflows work--precise to allow control over version requirementsI also realized that my Windows Updates vs Windows Upgrades analogy for cargo update and cargo upgrade breaks down a little because cargo upgrade can do "upgrades" that are on the level of cargo update (say we call it cargo upgrade --compatible). The difference is in the target audience (yourself vs your dependents)
cargo update only changes version requirements as a side effectThe primary role of cargo update has been to update your active dependencies (ie Cargo.lock). We do not plan to change that role but give the user control to force it to update in situations that were previously unsupported, particularly updating the Cargo.toml if need be.
Behavior:
cargo update --incompatible / cargo update -i will force (ie update version reqs) update unpinned, incompatible versions (no other dependencies)
--breaking or something else. The name depends on what expresses the concept clearly especially in light of any pinning behavior we havecargo update -p foo --precise ver will force the update to happen, only erroring if we can't (don't own relevant version reqs, is pinned), even if its incompatible but unpinned. Version requirement is only changed on incompatible.
Somewhere between deferred and rejected (speaking for myself): Support in cargo update for writing to the manifest for non-breaking changes, like bulk compatible upgrades of version requirements (ie a -save flag) which was one of our lower priority workflows. A --save flag is more about updating versions for your dependents, which while important for having valid lower-bounds on version requirements, doesn't fit with the existing model of cargo update. Maybe in the future we can find a way to express this in cargo update that fits with how it works or maybe another command can take on this role. We just aren't wanting to distract our efforts for handling most of the use cases to handle this one
While this tells a cohesive story, a part of me is somewhat concerned that this goes beyond the name update.
cargo update improvements--aggressive to --recursive (make --aggressive an alias for the new name)foo@x.y.z so that foo@x will work so long as its unambiguous (which it should be), much like cargo install foo@x.y.z supports auto-selecting the y and z fields (picks latest)cargo upgrade into cargo updatepackage argument to cargo update, removing the need for --package
0..=1 valuesThese are alternatives I had considered that help give an idea of what I mean by fitting into cargo.
cargo update always modifies Cargo.toml
We deprecate cargo update and a cargo upgrade always updates both files
We migrate to minimal-version resolution by default
cargo update becomes less useful and we move it out of the spot light.cargo upgrade is added that is focused on editing version requirementsSeparate commands for Cargo.lock (update) and Cargo.toml (upgrade)
apt dist-upgrade, maybe it could be cargo req-update?^ as we document them as being the same thing and sometimes people use ^ just becausecargo update && cargo update --incompatible vs cargo update --incompatible doing both as there are people who do want them separated and running two commands, while annoying, allows us to cover more use cases.FWIW, I do still care about this use case and use it regularly, and I would like to find a way to fit it into cargo. I'd be sad if we don't get this added as well. But I'm inclined to prioritize integrating cargo update -i, and if deferring this makes that possible sooner, I'm willing to defer it.