Pre-RFC: `cargo-script` for everyone

tools and infrastructure cargo
61

I would like to suggest the .rsh extension. It has .rs in it for the rust association but also sh which has the shell connotation. It appears unused by anything programming related.

3 Likes
62

Great, that sounds very reasonable!

63

I've updated the post with explorations of different embedded manifest formats, lockfile locations, and edition behavior. At the top is a link to the source document's history if you want to more easily see what changed.

I would ask that you focus the conversation first on the broader guidelines added at the start of "Rationale and Alternatives" section and at the start of each of the new sections. Settling on those first I think will help in discussing the details of the various options.

2 Likes
64

I've updated the post with explorations of different embedded manifest formats, lockfile locations, and edition behavior.

About writing the manifest as an attribute, the text now says:

  • Free-form rust code makes it harder for cargo to make edits to the manifest

I think this is equal or easier, actually. To edit a manifest embedded in an attribute:

  1. Tokenize the Rust source code (using any existing tokenizer, as long as it tracks spans and is aware of paired delimiters).
  2. Search the token stream for #![cargo(manifest = "...")] token sequences not occurring within any delimiters.
  3. Unescape the string literal token.
  4. Parse and edit TOML.
  5. Write out
    • the existing source text up to the beginning of the string literal,
    • the replacement TOML text as a literal (with suitable number of #s for escaping), and
    • the existing source text after the end of the string literal.

On the other hand, to edit a manifest embedded in documentation:

  1. Tokenize the Rust source code.
  2. Search the token stream for #![doc = "..."] token sequences (//! and /*! doc comment syntax turns into these) at the top level.
  3. Of these, look for ones that contain a ```cargo and start collecting text, inserting newlines.
  4. Parse and edit TOML.
  5. Write out
    • the existing source text up to the end of the first doc attribute that contained a ```cargo code fence
    • the replacement TOML text broken into lines and doc-comment-ified matching the doc-comment syntax used in the program
    • the existing source text starting from the beginning of the doc attribute that contained the closing code fence

These are more or less the same process except that the latter involves more concatenating and splitting, and deciding what doc comment syntax to emit. It might be easier to do the latter badly (that is, in a way which is unaware of Rust lexical syntax and may fail in edge cases), but the former is easy to do robustly, which I think should be a point in its favor.

3 Likes
65

I went ahead and prototyped something for this to see what it'd be like to use syn and friends to pull out an attribute. Editing a single string literal wouldn't be too bad (wish I had byte spans rather than line:column but oh well). Doc comments would be less fun to edit this way. I'll update the document.

#!/usr/bin/env cargo-eval

//! ```cargo
//! [dependencies]
//! clap = { version = "4.2.0", features = ["derive"] }
//! syn = { version = "2.0.14", features = ["full", "extra-traits"] }
//! quote = "1.0.26"
//! anyhow = "1"
//! proc-macro2 = { version = "1", features = ["span-locations"] }
//! ```

use clap::Parser;
use quote::ToTokens;

#[derive(Parser, Debug)]
struct Args {
    path: std::path::PathBuf,
    name: Option<String>,
    value: Option<String>,
}

fn main() -> anyhow::Result<()> {
    let args = Args::parse();
    let source = std::fs::read_to_string(&args.path)?;
    let file = syn::parse_file(&source)?;
    let Some(name) = &args.name else {
        for attr in &file.attrs {
            println!("{}", attr.meta.path().to_token_stream());
        }
        return Ok(())
    };
    let mut lits = Vec::new();
    for attr in &file.attrs {
        if attr.meta.path().is_ident(&name) {
            let syn::Meta::NameValue(nv) = &attr.meta else {
                anyhow::bail!("unsupported attr meta for {:?}", attr.meta.path())
            };
            let syn::Expr::Lit(syn::ExprLit { lit: syn::Lit::Str(lit), .. }) = &nv.value else {
                anyhow::bail!("only string literals are supported")
            };
            lits.push(lit);
        }
    }

    if let Some(value) = &args.value {
    } else {
        for lit in lits {
            let value = lit.value();
            let span = lit.span();
            let start = span.start();
            let end = span.end();
            println!(
                "{value:?} ({}:{}..{}:{})",
                start.line, start.column, end.line, end.column
            );
        }
    }

    Ok(())
}

$ ./attr-edit.rs attr-edit.rs doc
warning: unused variable: `value`
  --> /home/epage/src/personal/dump/attr-edit.rs:45:17
   |
45 |     if let Some(value) = &args.value {
   |                 ^^^^^ help: if this is intentional, prefix it with an underscore: `_value`
   |
   = note: `#[warn(unused_variables)]` on by default

" ```cargo" (3:0..3:12)
" [dependencies]" (4:0..4:18)
" clap = { version = \"4.2.0\", features = [\"derive\"] }" (5:0..5:55)
" syn = { version = \"2.0.14\", features = [\"full\", \"extra-traits\"] }" (6:0..6:69)
" quote = \"1.0.26\"" (7:0..7:20)
" anyhow = \"1\"" (8:0..8:16)
" proc-macro2 = { version = \"1\", features = [\"span-locations\"] }" (9:0..9:66)
" ```" (10:0..10:7)
66

About embedding the manifest, there are more options to reduce the work required in the compiler.

Why don't we specify the compiler to reject any rust file that contains the manifest syntax? Then we could specify the manifest to be specified as follows, on the top level of the root module:

#!/usr/bin/env something 

manifest {
    [dependencies]
    time = "1.2.3"
}

let current_time = ...;

For the compiler to parse this, it would need to understand toml enough to figure out what is the correct closing brace. But if it instead just aborts with a descriptive error message when encountering the keyword manifest, the integration would be very easy, wouldn't it?

The script execution utility could then strip the manifest out of the rust file before passing the stripped rust file to the compiler's stdin.

The advantage over all previous proposals is that this is the easiest to type.

Disadvantages are:

  • A new keyword. But if that is contextual it would still be possible to use it as identifier as freely as before. I don't know what is the policy on these contextual (hope that is the right term) keywords in the rust compiler, but if the compiler does nothing else than aborting when seeing it during parsing, then it should only have very local effects in its source code.
  • Script files cannot be copied into a cargo project without explicitly stripping the manifest away. This may be annoying, but then again it is fixed by just uncommenting the manifest or deleting it.
67

the parser wouldn't need to understand anything about toml, manifest could just be a proc_macro

macro_rules! manifest {
    ($($t:tt)*) => { /* macro magic */ }
}

manifest! {
    [package]
    name = "hello_world" // `#` doesn't work for comments
    version = "0.1.0"    // you'd have to use `//`
    authors = ["Alice <a@example.com>", "Bob <b@example.com>"]
}
1 Like
68

Macro was mentioned already in the pre-RFC. In addition to the drawbacks mentioned in there already, here are some more:

  • Compared to a keyword, you would need to add an extra exclamation mark. Not a big issue I guess.

  • It would be inconsistent as macros are usually used to generate code, and their output does not just delete the code. Feels a bit like a hack to me. Nevertheless, would be cheap to implement I guess?

  • As you demonstrated, you couldn't write plain toml anymore. That is a bit of a bigger issue in my opinion, as suddenly Cargo.toml and embedded manifest have different syntax.

69

One of the guidelines is that this needs to be first class

  • Every tool needs to be able to work with these files, rust-analyzer, third-party, etc. That is a lot of duplicated effort for each of these tools to strip the content
  • As called out, stripping the content before passing it to rustc is likely to lead to a second class experience when it comes to errors, etc unless we put in a lot of one-off work.

As called out in the guidelines, we should probably instead strive to use valid Rust syntax even if that syntax doesn't exist yet. This also means we need to be cognizant of mixing layers. The rust stdlib and language should likely not favor cargo.

Now, talking whether that should be the guideline is reasonable but I doubt this is one we can budge on.

2 Likes
70

So, maybe a weird idea, but what if instead of embedding the manifest in the Rust source, you embedded the source code in the manifest?

Currently you can define a hello world with Cargo.toml

[package]
name = "cargotest"
version = "0.1.0"
edition = "2021"

[[bin]]
name = "hello"
path = "foobar.whatever"

with the code in foobar.whatever:

fn main() {
    println!("Hello, World!");
}

So what if you could replace that with:

[package]
name = "cargotest"
version = "0.1.0"
edition = "2021"

[[bin]]
name = "hello"
inline-source = '''
fn main() {
    println!("Hello, World!");
}
'''

Of course, getting proper syntax highlighting would be that much more important and need tooling support.

One caveat would be that toml literal strings simply can't contain three single quotes in a row, but fortunately that's not very common in Rust source code.

71

My concerns over embedding source within the manifest are

  • This will have overhead for the "no manfiest needed" case when there should be none
    • If still defaulting fields in the single-file case, we'll have to key off of whether the file-stem is Cargo or not which seems less than ideal
  • rustc won't be able to compile this code directly, meaning that errors will be of a lower quality, not providing a first-class experience
  • The primary focus is on the rust code and so that should be given a higher quality experience (e.g. some editors have problems with syntax embedded in syntax)
1 Like
72

Some comments on the edits:

  • Passing flags in a #! doesn't work cross-platform

(this was in the context of configuring embedded lockfiles via command line flags)

Cargo already has to read the file for the manifest, it can just continue reading until it has either also found the lockfile, or reached the end. If it finds a lockfile, then it can just use it. So passing flags to #! isn't required, as you configure it when invoking the file, but then the fact that a lockfile is embedded into the file is just simply stored by the fact of there being a lockfile. This also prevents the case where you have a separate lockfile and the embedded one gets outdated because cargo doesn't update it any more.

Also, IMO the RFC should list the suggestions for size optimizations of the lockfile, as lockfiles can get big: either listing the pairs of version numbers plus name (instead of encoding the entire graph), or timestamp (with the caveat that such a "lockfile" doesn't support per-package upgrading, either it's a clean cargo update output, or it doesn't work). Maybe also one could consider a combined strategy: the timestamp serves as basis and then one has a list of "commands" to modify that set of (package, version) tuples, simulating cargo update -p behaviour. This isn't without disadvantages, but at least it should be mentioned in the RFC.

Also, I suggest to include my suggestion for specifying the manifest as a comment. To explain it again, if the first line after the shebang starts with a /*, then the comment that is inside is the manifest. Comments after that are not part of the manifest. This is not rustdoc compatible, but it saves a few characters compared to needing to make a source code listing inside rustdoc comment. The logic is simple enough that syntax highlighters can support it (I hope :)).

73

A small proposal, perhaps this could work well being called cargo-rush and use a .rush extension. The name is snappy and extends the shipping metaphor. There’s a crate that was yanked 8 years ago called rush and a rust shell project named rush however.

Regarding the syntax, another option would be to go pure rust instead of flipping to toml (you want toml, import the toml crate and use the toml! macro.

Would one of the following options be the craziest thing in the world?

#!/bin/env cargo-rush

cargo_rush::Manifest::default()
  .dependencies(vec![
    (“time”, “0.1.12”),
    Dependency::from(“clap”)
      .version(“4.2”).features([“derive”),
  ]).init();

use time::Time;
fn main() {
  // ...
}

Or

#!/bin/env cargo-rush

cargo_rush::Manifest::default()
  .dependencies(vec![(“toml”, “0.4.2”)])
  .init();

cargo_rush::Manifest::from(toml!(
  [dependencies]
  time = “0.1.12”
  clap = { version = “4.2”, features = [“derive”] }
).init();

use time::Time;
fn main() {
  // ...
}

Or perhaps something like:

#!/bin/env cargo-rush

static TOML: cargo_rush::Manifest = cargo_rush::use(("toml", "0.4.2"));
static RUSH_MANIFEST: cargo_rush::Manifest = TOML.toml!(
    [dependencies]
    time = “0.1.12”
    clap = { version = “4.2”, features = [“derive”] }
  ).into().init();

use time::Time;
fn main() {
  // ...
}

In some recent related java discussions, Brian Goetz wrote Paving the on-ramp with the goal of making the simple cases easy to write for teaching java. It mentions JEP 330: Launch Single-File Source-Code Programs, and was discussed in a thread on the amber-project lists in September 2022, which continued into October 2022. There was a JEP writeup at JEP 445: Flexible Main Methods and Anonymous Main Classes (Preview) discussed in Feburary 2023 on amber-spec-observers. The threads cover some perspectives that could be useful to take note of in implementing this, mainly focused on teaching, but I think some of the conceptual parts of the threads are probably relevant here too.

1 Like
74

One more suggestion I have is around future support of build.rs and maybe even proc macros inside the same file. Note that I don't think that these should be part of the initial release, but it would be great if syntax could be reserved e.g.

#!/usr/bin/env cargo-eval

fn main() {}

// cargo-eval-file: build.rs
mod build { /* ... */}

would indicate that the content of the module would be the content of the "build.rs" file. Unlike in Rust, these special modules would only be allowed on the top level. There are probably syntaxes currently forbidden by Rust, like #[cfg(file::build_rs)], so one isn't out of options when not doing anything, but IMO it's good to at least do some initial brainstorming on it.

Edit: issue: non-main file future compatibility · Issue #153 · epage/cargo-script-mvs · GitHub

1 Like
75

A variant of this idea (inside an attribute instead of a root-level item) was covered earlier. Since then, the RFC was updated to include these comments

Single-file packages should have a first-class experience

  • Easier unassisted migration between single-file and multi-file packages
  • Example implications:
    • ...
    • Manifest formats should be the same rather than using a specialized schema or data format Friction for starting a new single-file package should be minimal
  • Easy to remember, minimal syntax so people are more likely to use it in one-off cases, experimental or prototyping use cases without tool assistance
  • Example implications:
    • ...
    • See also the implications for first-class experience
    • ... ... As an alternative, manifest could a less stringly-typed format but that makes it harder for cargo to parse and edit, makes it harder for users to migrate between single and multi-file packages, and makes it harder to transfer knowledge and experience

I'd recommend we focus on these and the other guidelines before talking specifics of the trade offs of syntaxes like this.

76

btw I found a fairly large list of prior art in my backlog of tabs...

I'll be adding this to the document and noting particular prior art I find in it

77

This was covered as

Configuration 4: User-created empty lockfile

The user could drop an empty lockfile in the agreed-to location and cargo-eval could detect that and use it.

Maybe the wording can be used. I had these framed in terms of bootstrapping the package.

Updating it to

Configuration 4: Exitence Check

cargo-eval can check if the lockfile exists in the agreed-to location and use it / update it. To initially opt-in, a user could place an empty lockfile in that location.

Also, IMO the RFC should list the suggestions for size optimizations of the lockfile, as lockfiles can get big: either listing the pairs of version numbers plus name (instead of encoding the entire graph), or timestamp (with the caveat that such a "lockfile" doesn't support per-package upgrading, either it's a clean cargo update output, or it doesn't work). Maybe also one could consider a combined strategy: the timestamp serves as basis and then one has a list of "commands" to modify that set of (package, version) tuples, simulating cargo update -p behaviour. This isn't without disadvantages, but at least it should be mentioned in the RFC.

For me, most of this falls into the "second class" experience. Some of this also leads to a lot of complexity and slow downs (command log would require each run to do multiple resolves just to check the cache)

Also, I suggest to include my suggestion for specifying the manifest as a comment. To explain it again, if the first line after the shebang starts with a /*, then the comment that is inside is the manifest. Comments after that are not part of the manifest. This is not rustdoc compatible, but it saves a few characters compared to needing to make a source code listing inside rustdoc comment. The logic is simple enough that syntax highlighters can support it (I hope :)).

I'm adding it (also looks like dlang's dub, kotlin and gorun do something similar).

1 Like
78

This would be extremely helpful for bootstrap/x.py and I think would allow us to get rid of the x and x.ps1 shell scripts altogether.

1 Like
79

Isn't the idea of using a python script also that you don't need a rust installer tool chain pre-installed at all in order to build rustc yourself? I feel like any form of cargo-script wouldn't achieve the same.

Maybe I'm misunderstanding your point though, as you don't mention getting rid of x.py itself.

80

Most of the logic of the x build tool is already actually in Rust; the python/shell entry point is primarily just in charge of downloading a seed toolchain to compile the main chunk of the tool. If you do just --help it'll even warn you that it needs to compile before printing the actual help message.

In theory x could be made runnable via the cargo-script interface, but this does still carry the issue of needing a recent enough system installation of Rust to seed that from even if the compiler seed is downloaded later, and the script entry point would still be desirable to have so that bootstrap can be completely self contained rather than requiring a separate prerequisite step.

The shell shims are just to best effort grab the correct python and use it to launch x.py; these would be less useful if the easy way to run x was via a preexisting Cargo, even if the x.py wrapper stays.

2 Likes