Cargo's next few years

[nrc]

As part of the roadmap process for 2019, the Cargo team have been thinking about the next few years. I’ve written that up as a blog post: https://www.ncameron.org/blog/cargos-next-few-years/

[JeanMertz]

Great post @nrc, I was pretty much nodding all the way through.

I was happy to see the “2019 again: plugins” part at the end of the post. In general, Cargo is one of the best tools in the Rust ecosystem, and one that I have the least complaints about (although yes, cross compilation would be great to have), but the discovery of Cargo plugins is an issue, and prevents me from leaning too much on them in my projects. I would like there to be some kind of system that either a) makes it possible for a project to add cargo plugins as requirements for the project, or b) suggest installing a plug-in whenever the user runs a command that requires that plugin.

I had one (big) confusion while reading the post:

You start the post with:

The Cargo team have been thinking about and discussing long-term plans for Cargo. In this post I’ll talk about what we hope Cargo will look like

But for the rest of the post you talk in the “I” form:

Over the next few years I don’t want to prioritise

Instead I want to focus on

I think we should focus on one group of users at a time

I propose we focus on the following domains

So am I reading your proposal on the future of Cargo, or the consensus of the Cargo team, which happens to match your thoughts exactly?

[matklad]

a) makes it possible for a project to add cargo plugins as requirements for the project,

Cargo workflows (http://aturon.github.io/2018/04/05/workflows/ ) were a major theme during the last all-hands and were sort-of planned for 2017-2018, but that didn’t pan out. I personally feel that allowing ‘cargo foo’ to be project local and be independent of the global system state is somewhat more important than making writing cargo-foo easier.

EDIT: though the alias trick(https://matklad.github.io/2018/01/03/make-your-own-make.html) works for my use cases

[matklad]

Another topic which I would love to see on the roadmap is solving the feature selection.

This is a very technical issue in one of the core abstractions of Cargo: feature selection is done globally per “compilation session” and not locally, per compilation target. User visible artifacts are

• just adding a dev dependency activates features for main binary/library
• different binaries in different packages of the workspace affect each other’s features
• In a workspace, cargo build -p foo and cd foo; cargo build produce different results.

Fixing this is hard, could break existing builds if done without care (and “care” here could amount to maintaining two implementations of feature selection) and will not immediately improve life of the users.

Not fixing this seems pretty bad as well: bugs in core abstractions are the worst kind of debt.

It may also be the case, given all the constraints, that the best resolution here is “won’t fix”.

I think the latest discussion of one aspect of this problem is this: https://github.com/rust-lang/cargo/issues/5364.

It might be a good idea, in this relatively uneventful for cargo period, to alocate some time to this family of problems.

[djc]

My most salient thought about this when I was reading the blog post this morning was that this feels like it’s missing “the trees for the forest”: that is, this focuses on longer-term large features, when in my mind, there’s a substantial amount of tech debt in the cargo code and UX that makes cargo harder to work with as soon as you get slightly off the beaten path. Some of this might be addressed by the plugins that the OP talks about for this year, but I’m not sure that would be enough to get a solid base to build all that other stuff on top of.

I really liked @briansmith’s take on this on Reddit, and I also read @matklad’s post as supporting this idea.

(And see also the Cargo section from my Rust 2019 post.)

[kornel]

I second that. Configuration is a major problem.

Cargo’s features are also too limited and awkward to use for dependencies-of-dependencies. Because of that -sys crates resort to using env vars instead, but that brings whole another bundle of problems: Cargo doesn’t track env vars, and doesn’t have a place to store them. Env vars are global and affect the entire build, so they can break proc macros and cross compilation. Env vars feel fragile, undiscoverable and ad-hoc.

I’d love Cargo to properly handle more of crate config “natively”, including ability to combine disto-specific, platform-specific, project-specific and user’s preferred configuration:

Issue: There's no way to make `cargo build` - with no env vars - build redistributable executable

[nrc]

My most salient thought about this when I was reading the blog post this morning was that this feels like it’s missing “the trees for the forest”: that is, this focuses on longer-term large features

This was intentional! The post is meant to be about the big ticket items and long-term path for Cargo. There is of course a whole lot of ‘small’ things that need to be done in the meantime. We’ll have another post about specifics for 2019 in a week or two.

So am I reading your proposal on the future of Cargo, or the consensus of the Cargo team, which happens to match your thoughts exactly?

Hmm, yeah, the pronouns worked out a bit weird. This post is the consensus of the Cargo team and the ideas belong to the team as a whole. The write-up is mine, but reviewed by the team.

I personally feel that allowing ‘cargo foo’ to be project local and be independent of the global system state is somewhat more important than making writing cargo-foo easier.

@matklad could you expand on what that means please? It sounds like it ties in with our focus on plugins, but I’m not exactly sure what you want.

Another topic which I would love to see on the roadmap is solving the feature selection.

Definitely! Revisiting features will be on our roadmap for 2019, I didn’t mention it here because we consider it part of addressing technical debt, rather than doing something new.

[matklad]

@matklad could you expand on what that means please?

Aaturon’s blog post is a good description: http://aturon.github.io/2018/04/05/workflows/

The gist is adding a [task] section to Cargo.toml with the same syntax as dependencies, and making a cargo task foo command, which finds and runs a foo binary from a task package. The crucial properties which make this different from Cargo subcommands is that tasks are local to a workspace, locked via a lockfile, and work out of the box (no need to add "run cargo install cargo-foo" to your project’s readme). This unlocks the following use-cases:

• writing a “small-scale” automation, which is typically written in bash, in Rust, such as it is cross-platform, has access to powerful libraries like serde, and has tests
• easy distribution for framework-specific automation: diesel might have a diesel-tasks package which provides cargo task generate-migration
• allow to bootstrap a truly generic build system (a-la make/gradle) from Cargo. Project like just or cargo-make exist, but the main problem with them is that using them is not reproducible: you need to add "run cargo install cargo-make" to the readme, and hope that future users will get a sufficiently compatible version. cargo task make will be more or less inflatable.

It sounds like it ties in with our focus on plugins?

Maybe? From the blog-post, I gathered that the main focus is on making writing custom Cargo sub commands easier, by expanding the public API Cargo offers. This seems pretty challenging to me, stability wise (see two awesome diagrams about plugins here). What personally would be more interested in is ability to run custom sub-commands as a part of reproducible build.

[nrc]

I think the idea of cargo task is great and something we might want to pursue (note that this really comes under the ‘workflow’ heading, so it might not happen soon - I think that is ok - there are a lot of open questions around this thing). What is more interesting to me is how these tasks are written and to what level Cargo understands them. I think if people writing the tasks have to replicate Cargo in various ways, then we just end up with a worse version of today’s plugins.

writing a “small-scale” automation, which is typically written in bash, in Rust, such as it is cross-platform, has access to powerful libraries like serde, and has tests

we certainly want to achieve something this - in the post I talk about scripting and workflows and I think it is an important goal in the quest to make Cargo more flexible. But there are lots of pressures - if the tasks are just Rust, then you end up with problems like build scripts have in terms of making parts of Cargo re-usable or in knowing when to rebuild.

easy distribution for framework-specific automation

again, this is a goal of the scripting/workflows stuff, but the question is so open I don’t think we should prioritise it now. Instead we should do more work to understand the requirements and constraints in this space.

allow to bootstrap a truly generic build system (a-la make/gradle) from Cargo

This is a non-goal for the Cargo project.

the public API Cargo offers. This seems pretty challenging to me, stability wise

a couple of examples I can think of is extracting a Cargo.toml parser or a Cargo metadata parser and serialiser. Both those things exist today but are not used in Cargo, so there is duplication of code and all that goes with it. I would like to make official crates which are used by Cargo and maintained by the team so plugin authors can be sure that their plugins will have the same behaviour as Cargo and not break with new versions.

[matklad]

allow to bootstrap a truly generic build system (a-la make/gradle) from Cargo

This is a non-goal for the Cargo project.

Looks like I’ve failed to be clear enough I don’t want Cargo to grow make-like capabilities. However, I want to make it possible for someone else to write a rusty-make thingy, which can be used by Rust projects, without installing it.

I think if people writing the tasks have to replicate Cargo in various ways, then we just end up with a worse version of today’s plugins.

Why worse? Reproducible builds and “you don’t need to mention install step in project’s README” seem to me clear wins for the users of the system, everything else is the same.

a couple of examples I can think of is extracting a Cargo.toml parser or a Cargo metadata parser and serialiser.

Oh, this is much less risky that what I’ve read from the blog-post Making existing public APIs easier to use is always good! I think I was alarmed by terminology switch from “custom Cargo commands” to “Cargo plugins”.

[nrc]

Why worse?

Because not only does every plugin potentially do something different to Cargo, but every project you try to build might!

[Ixrec]

I want to make it possible for someone else to write a rusty-make thingy, which can be used by Rust projects, without installing it.

I think there’s a word choice issue here because “using” a program without “installing” it sounds like a contradiction to me. What did you mean?

Also, obligatory mention of just

[Nemo157]

I believe what @matklad is referring to is globally installing the tool. Instead, a project could have additional “task” dependencies that are built into the projects local target folder as needed, and could be correctly versioned in the Cargo.lock.

For example at the moment you must use the exact same version of wasm-bindgen in your project as the globally installed wasm-bindgen-cli, if you have multiple projects using different versions of wasm-bindgen you have to keep reinstalling different versions of wasm-bindgen-cli (or juggle managing multiple installs on different paths). If instead you had wasm-bindgen-cli declared in your Cargo.toml and invoked it via cargo somehow, you would have per-project versioning handled automatically, and new developers of your project wouldn’t need to know to run cargo install wasm-bindgen-cli --version <whatever>.

[matklad]

I think there’s a word choice issue here because “using” a program without “installing” it sounds like a contradiction to me. What did you mean?

If you have

[tasks]
just = "1.0.0"

in Cargo.toml, than workflow for the contributor is

$ git clone github.com/foo/bar && cd bar
$ cargo task just x

instead of

$ cargo install just  # global state, everything can go wrong here
$ git clone github.com/foo/bar && cd bar
$ just x

That is, tasks solve bootstrapping problem: you don’t need to dump jars to the repo to make ./gradlew work, you don’t need to “make sure you have Python in you $PATH” to make ./x.py work. Everything is bootstrapped from Cargo.

[notriddle]

Yes! Yes, yes yes yes yes!

Project-local tasks are my biggest gripe about Cargo compared to Mix. The lack of library-provided cookie-cutter code generators and platform-specific package generation in a way that’s pinned to the project (cargo install, as has already been mentioned, is global) is annoying after you’ve experienced better.

[kornel]

Project-specific tasks work OK in npm. They’re not perfect, but on the right side of 80/20. It may also be a good playground to prove which kinds of tasks are useful and common enough to warrant inclusion in Cargo.

[ahmedcharles]

I’d suggest that the idea of having plugins/tasks which are versioned and project specific should be orthogonal to how they are developed/written. By that, I mean, the following should have the same result:

cargo install tree
cargo tree

cargo task tree

This has the advantage of turning an m * n problem into an m + n problem. Making it easier to write cargo-tree is a worthy goal but so is having users be able to use a consistent version of cargo-tree in a transparent way.

[CAD97]

Definitely the Cargo “API surface” they they work against can be standardized. I’d be surprised at anything else.

I think the split between tasks and subcommands should be applicability though. Subcommands are really best when they’re tools for the developer (like cargo-tree or cargo-outdated). These are “global” capabilities added to cargo that don’t need to be fully standardized between developers working on a project.

On the other hand, tasks are great for the project-related workflows. Something like cargo-make or any bootstrapping setup will be required to be semver-compatible on all developer machines to produce a consistent result (and shared lockfiles would of course mean exactly compatible).

There is a difference between when you “want” to use the two, based on whether it is a global capability or something that needs to be shared between developers of a project. But yes, it’d also be great if the same tooling can “just” work both as a global subcommand and as a local task.

[mark-i-m]

cargo task foo sounds great, but I would really like to see some sort of sandboxing for build scripts and tasks for security before we go down that path…

[petrochenkov]

To clarify, aren’t “project-specific tasks” equivalent to custom actions/commands as they are found in various build systems?
I mean “edges” in the dependency graph of build targets.

If we get the “edges” then we’ll only need “nodes” to turn Cargo into a proper build system.

I had a thought a couple of times that it would be nice for Cargo to support “virtual crates” (aka custom targets) in its manifest as the “nodes” in its dependency graph, whose only purpose is to run their build.rs (aka custom commands) when their dependencies are out-of-date.