A replacement registry may omit versions. In addition versions may be entirely removed for legal reasons as opposed to just yanked.
1 Like
Right, I was also anticipating that there might be some problems with that and how the registries work. I guess my point is just that there only needs to be enough information to choose the right version from a set of versions, in a reproducible manner, and this means in principle that more compression can be achieved.
I wasn’t sure about the precise way in which versions removed for legal reasons operate in the first place; things I’ve wondered included “if a crate version can be removed for legal reasons, can a new version with the same semver string be (re-)published later?” In which case reducing the lockfile entry to the full crate name and version string along seems questionable, too, and one maybe really wants the hash of the source code included.
The following idea may be completely stupid, I’m that familiar with how crates.io stores its data internally
From what I understand of the recent sparse registry option that is currently being stabilized, this probably means that crates.io knows when every updates have been made (I assume adding a new version into crates.io is just adding new commit in an internal git repository). If that’s indeed true, then the embedded lockfile could simply be a single date (including the second), and during dependency resolution we would ignore any revision newer that this date, making this process effectively deterministic as long as the date is not in the future.
3 Likes
That would resolve most of the "minor version upgrade broke my package build" concerns. It's an interesting proposal I think. Technologically it's possible to pull it off via adding the publication time of crate versions to the index (as checking out an older git commit won't work as the git history is regularly squashed and git is being migrated away from), so in theory one can make it work. It won't be a 100% replacement for lockfiles as they can have more states than just "clean cargo update as of day x time mm:ss", you can also do stuff like cargo update -p <pkg-name> --version. But at least many users would be way better off with this than without having any lockfile support at all.
I agree that inline lockfiles should be opt-in. I would propose the following logic:
- if there is no inline lockfile, cargo doesn't add one
- there is a flag to add an inline lockfile to files, say what you proposed
cargo eval --savefor example, or maybecargo eval --save-lock. - if there is an inline lockfile but it is outdated, cargo tries to write the file with an updated inline lockfile (just like it's done with Cargo.lock today). If the write fails, it issues an error but there is a flag to turn off the error.
But the default behaviour should be to put the .lock next to the .rs. It's not beautiful to look at, yes, but I think it's a fair price to pay for having dependencies. If people have a directory with multiple scripts the suggestion is to make a workspace with a shared lockfile. So the number of lockfiles per directory should be low.
Since I've made my earlier comment I've also realized that compactifying lockfiles also has a cost associated to it: git merge conflict risk. The lockfiles used to have a compact section listing all the hashes of all present crates, but then the format was changed to move the hashes into each dependency's section. This helped with reducing the number of merge conflicts as multiple people edited the same Cargo.lock file in PRs. However, I don't think this issue is really a problem for single-file scripts: Cargo.lock files of large repos with tens of thousands of lines with many collaborators are obviously seeing way more conflicting changes to them than an one-off 100-line script. And if the inline lockfile really causes problems to people they can only use them for distribution purposes of the script (when offering downloads of it).
If the lockfile is not inlined into the file, why should it be put into the same folder as the .rs and not into CARGO_TARGET_DIR ?
If it is put in the target dir you lose the lockfile every time you do cargo clean. It prevents checking it into a vcs and it also puts it out of the view of the user, making it easier to forget shipping it together with the script itself.
1 Like
Furthermore, it doesn't work outside a cargo project (eg in ~/bin or whatever) - which I think is the more interesting use case for this design (I would prefer a scripts dir using the project/workspace lockfile for project scripts)
No, but you can read the shebang line yourself and run with the arguments you see instead of what you actually got. If you want to go down that rabbit hole, here's an entrance to poke at.
This is actually easier with git because the index gets updated atomically but not with the sparse registry (yet).
However, before people latch too much on to this idea, resolution date is insufficient. Cargo only partially resolves when each new dependency is added and then we also have cargo update --package and --precise. We'd have to keep a lockfile update log (what dependency requirements changed at what resolution timestamp) which is starting to get quite complicated.
There is also runhaskell for Haskell code.
1 Like
@matklad I'm currently enumerating trade offs for a different file extension. From rust-analyzer's perspective, what are the trade offs of .rs vs something different?
- I could see rust-analyzer treating editing of all
cargocode fences the same, so no unqiue extension is needed for this - I assume rust-analyzer would need to map a source file back to its manifest for running commands (
cargo run,cargo check, etc). As#!is unix-like specific and not relevant to Windows, I doubt we want to keep off of that.cargocode fences aren't required. This is making me suspect we need a unique extension but I wanted to double check first.
I've updated the post with my latest edits and included a link to the commit range for people to see how it changed. I've still not processed the feedback around editions, lockfiles, and the format of the embedded manifest.
The challenge with cargo-exec is how to distinguish it from cargo run. I chose "eval" to mean "evaluate this file". I can see the overlap with REPL semantics though.
In reading up on some other prior art I was pointed to, I thought it was interesting that cabal uses the top-level binary for #! and then uses cabal run for everything else. I was concerned with ambiguity between subcommands and scripts but on at least my system, the script path is canonicalized so we should be able to clearly tell the difference between the two.
This feels quite nice:
#!/usr/bin/env cargo
fn main() {
}
The downside isit would make passing in compiler flags a bit verbose and maybe non-obvious
#!/usr/bin/env -S cargo run --release --manifest-path
fn main() {
}
There should be something that signifies “you can run this”, both for humans and for ra, to distinguish between scripts and normal rust code which would be included elsewhere.
Either extension, or some sort of preamble inside the file would work. Extension would be marginally easier.
I think this is a great idea and it would really be useful to allow Rust to be used as a "scripting language".
However, the current proposal could be improved by studying the potential real-world use cases of a scripting language in more detail. For example, the use case of a programmer who writes a lot of python scripts to achieve simple tasks. I personally know that this happens a lot in computational sciences. There are probably also other fields where people use python to write small programs.
With python, libraries are installed globally on your machine and you do not have to declare them as dependencies within your file. Instead, you simply import them. This would be equivalent to a rust script just containing use statements. The actual versions could be stored in the user's .cargo directory instead of in any single script. With python, libraries are installed globally using e.g. pip. In Rust, there could be a similar subcommand of cargo that lets the user select versions of certain libraries. Any invocation of a Rust script would then use these versions.
With python, if one does explicitly want to share a script including its dependencies, usually one shares some environment file as well. This would be like sharing the script file and additionally the Cargo.lock in Rust. There are also commands for e.g. pip to export or import and environment to or from a file, such that one does not have to write the environment file manually, but can just note down the local environment. In Rust, this could be achieved by similar commands for cargo.
I would call this approach used with python the "mess-by-default" approach, because scripts by default are missing important information for how they should be compiled, so compiling the script on one machine will likely not yield the same result as compiling it on another. The approach that was discussed here so far with explicit dependencies stated for each script could be called the "rusty" approach.
The mess-by-default approach makes it easier to write scripts. One only needs use statements, and does not need to manage library versions for each single script. Also, one does not need to remember which version of some library one is using, which may be different from the latest major version. And if someone has ten scripts using the same library in some project, writing the version every time will likely feel redundant fast. Lastly, anyone who is now using python will face a barrier when switching to Rust scripts if they have to add detailed dependency specifications in each script.
All that being said, I believe that mess-by-default and rusty are not in a contradiction with each other. The mess-by-default approach could be the fallback in case the script file does not contain a manifest, or uses some crate that is not mentioned in the manifest. With proper tooling, one can also fearlessly start with mess-by-default, and as soon as there are problems one can automatically convert to the rusty approach.
On a different note, about the file extension of a rust script: How about .srs? It contains the sequence rs that is used for Rust files, and the letter s that indicates a "script", and is not already used by a well-known technology like .rss.
Also, as some other people have already mentioned, for the option to include a manifest inside a Rust script, it would not be the best choice to use comments. My arguments to that are that comments are meant to be for humans, and should not be required for the compiler to compile the code correctly. I think that one should be able to strip a program from all comments, and then still get the same result from a compilation.
I believe that the approach of using #![...] is nice, since these annotations usually tell the compiler how to compile something on a more abstract level, like generating some extra code using a macro or conditionally hiding some code under certain circumstances.
Unix generally denotes this with a #! preamble and ignores the file extension, and Windows generally relies on file extensions for such things. To my mind, the best solution is for RA to recognize a file with either marker as a script-like program and let each OS rely on the mechanism it traditionally uses.
To write a cross-platform program, you'd need to use both markers. This seems quite common for scripting languages like Python.
1 Like
T̶h̶e̶ ̶-̶S̶ ̶m̶i̶g̶h̶t̶ ̶b̶e̶ ̶e̶a̶s̶y̶ ̶t̶o̶ ̶f̶o̶r̶g̶e̶t̶,̶ ̶a̶s̶ ̶i̶t̶s̶ ̶n̶o̶t̶ ̶u̶s̶e̶d̶ ̶i̶n̶ ̶c̶o̶m̶m̶o̶n̶l̶y̶ ̶u̶s̶e̶d̶ ̶s̶c̶r̶i̶p̶t̶ ̶s̶h̶e̶b̶a̶n̶g̶s̶.̶ ̶A̶ ̶s̶h̶e̶b̶a̶n̶g̶ ̶l̶i̶k̶e̶ ̶̶#̶!̶/̶u̶s̶r̶/̶b̶i̶n̶/̶e̶n̶v̶ ̶p̶y̶t̶h̶o̶n̶3̶̶ ̶i̶s̶ ̶s̶e̶e̶n̶ ̶a̶ ̶l̶o̶t̶ ̶f̶o̶r̶t̶u̶n̶a̶t̶e̶l̶y̶.̶ ̶
Cargo/RA could inspect the first line. If it finds a shebang it extracts the arguments from the shebang. T̶h̶a̶t̶ ̶w̶o̶u̶l̶d̶ ̶a̶l̶l̶o̶w̶ ̶u̶s̶e̶r̶s̶ ̶t̶o̶ ̶f̶o̶r̶g̶e̶t̶ ̶t̶h̶e̶ ̶-̶S̶ ̶(̶a̶n̶d̶ ̶p̶o̶s̶s̶i̶b̶l̶y̶ ̶i̶s̶s̶u̶e̶ ̶a̶ ̶w̶a̶r̶n̶i̶n̶g̶?̶)̶.̶ For .rs (or another extension) files we would add a cargo run/eval/script action in the windows context menu.
I do not see another way to pass cargo arguments on Windows.
Edit: thanks @epage for correcting me
My understanding is that without -S, the whole string is treated as argv[0] which will fail, so we can't add arguments to later be processed within the #! line unless we take the jbang route of avoiding #! with ///usr/bin/env jbang "$0" "$@" ; exit $?
1 Like
Before I get to my downer points, I will say that this addresses my concern of how to allow sharing of dependency versions between low-effort scripts to reduce build times.
With that said, I suspect a lot of Python developers (me being one) feel like the global environment is flawed and exclusively use venvs. If nothing else, you can run into version conflicts.
This can also be emulated with the currently documented future possibility of including these scripts in workspaces.
Weighing those out along with the workflows around sharing scripts, I see the embedded manfiests as the minimum solution we should start with and then we can iterate and develop this further. Third-party subcommands are a great way to experiment with workflows like this before getting to the RFC stage, much like cargo-script and its variants have existed for a while.
4 Likes
Thanks a ton for working on this! Even in languages that have REPL and single file executability already, there are tools like bundle-inline (for Ruby) that allow for managing dependencies in a single file. I think it's helpful to look at other ecosystems and see what they're doing in the space:
$ mkdir test-bundle-inline
$ cd test-bundle-inline
$ vim scratch.rb
$ cat scratch.rb
require 'bundler/inline'
gemfile do
source 'https://rubygems.org'
gem 'json', require: false
gem 'nap', require: 'rest'
gem 'cocoapods', '~> 0.34.1'
end
puts 'Gems installed and loaded!'
puts "The nap gem is at version #{REST::VERSION}"
$ ruby scratch.rb
Gems installed and loaded!
The nap gem is at version 0.8.0
$ ls
scratch.rb
It hides the installation by default, and the bundle inline tool doesn't handle the lockfile generation. If you want to lock things down, you must declare it explicitly, though dependencies in Ruby are global.
I like the idea of a single script Rust file also being a regular Rust file either via a cargo TOML comment or through a more language-like interface.
I do think that reproducibility is are really important feature in the Rust ecosystem. One thought I had for handling lockfiles could to allow the script to modify the source code and embed them in the file as comments. It would have to handle contention with other parallel execution, likely through an OS lock on the file at resolution time (I'm not sure if windows has a FLOCK like feature?). It would be unexpected for a script to modify itself, though and the lockfile contents can be quite long/large. So possibly putting it at the bottom of the file. This feature may be off by default and enabled via flag, env var, or specific comment.
Shipping something without solving the lockfile case is still useful, though. As epage already linked, we're wanting to use Rust even in cases where we might otherwise write a bash script. So even a rust script that sometimes fails due to dependency issues feels safer to write than a bash script.
1 Like