What do Rust tools need from the build system?

tools and infrastructure
1

As part of the work on build system integration, the Cargo team is collecting information about what Rust tools need from a build system.

Today, tools largely assume and use Cargo for these purposes. Our ultimate goal is to provide a single, uniform interface – perhaps through Cargo – that tools can rely on for all of their build system needs, but that will interface with external build systems as needed, without the tools having to know anything. In other words, we want Rust tools to “just work” regardless of how the Rust project is integrated into a larger build system context.

But to do this design, we need to fully understand the needs of Rust tools. That’s where you come in!

Please have a look at the above doc and add your thoughts, liberally.

2 Likes
2

Don’t have a DropBox account so I’m chiming in here:

libenclave-tools

  • Pre-build: determine value of OUT_DIR of some dependency for a particular build invocation (alternatively: build tool and dependency build script need a way to coordinate on linker command line)
  • Post-build: find build artifacts
1 Like
3

Cargo should support CI systems in the same way java maven does ( https://maven.apache.org/ ). It should be easy to integrate with tools like Jenkins.

CI servers should have opportunity to add/remove libs, repositories, change paths, etc without modify the .ml files. It should be easy to build/test artifacts for different architectures again without modify TOML files.

Secrets (like ssh passwords, AWS certificates, etc) should be managed by external tools and Cargo should let this behavior. It is normal for example that a CI server deploys built artifacts in some servers and that CI provides certificates to access to them. Again, a good source of inspiration is maven (check settings.xml and security-settings.xml and how maven uses them)

A good pattern to follow is maven integration with CI tools. A good antipattern is Scala SBT ( http://www.scala-sbt.org/ ). In SBT is impossible to change the build without change the build script (literally a program). SBT is not enterprise friendly.

UPDATE It should be easy to integrate Cargo with matrix builds (a combination of builds by architecture/platform/features. Check https://wiki.jenkins.io/display/JENKINS/Matrix+Project+Plugin

2 Likes
4

Not sure if/how relevant, but I package rust and cargo for a source-based Linux distribution, Gentoo Linux. For us, things like being able to extract a set of dependencies, in the sense of URLs to actual packages, as well as an easy way to propagate build-time flags and features. Maybe also a way to (make it easier to) externally generate and then use a Cargo.lock file. Another thing is that it would be nice if Rust’s LLVM build was more tightly scoped, for example IIRC it now always builds all backends, even when a rustbuild config only specifies one backend.

One shortcoming in cargo that I really dislike based on how we use feature flags in Gentoo is that there is no way to disable some feature without disabling all features. It seems to me that the crate author should specify default features, and the default mode for people to use that should be to add or subtract features against that baseline of the default feature set. I think it would be a bad experience if, for example, I want to switch off features a and b but keep feature c. Now, the author would maybe like to make feature d optional-but-on-by-default (for compatibility reasons), but I will never get d because I had to disable all default features in order to disable a and b.

It would also be good to have better support for code coverage. In particular, I have not found a good way to generate coverage data for derive-based procedural macros (because these get dynamically loaded as part of the compiler, I guess).

3 Likes
5

One of my plans for Rustbud is to have a list of tools to cargo install into a build environment. To make that reproducible, I need a way to take a version specifier (like xargo = "^1.2.3") and resolve that to a specific version (like “1.3.4”) that I can put in my lock file. This is exactly the same thing Cargo already does when building its own lock file, except that these kinds of things don’t belong in Cargo.toml because they’re not strictly speaking dependencies of the crate in question.

I could do it myself by poking at the crates.io API, of course, but Cargo is getting fancy new alternative-crate-repository features, and I’d rather not re-implement all those.

I imagine it working something like this:

$ cargo resolve xargo '^1.2.3'
1.3.4

EDIT: Wait, no, that’s not going to work. I’d have a circular dependency: I’d need an environment with Cargo to resolve versions so I can write my lock file, but I’d also need a lock file to create the environment with Cargo in it. Nevermind, I’ll have to think of something else.

6

(I can’t edit dropbox)

For cargo-deb I need:

  • Most of standard Cargo.toml metadata like description & readme file + custom metadata for cargo-deb
  • List of all binary targets built.

cargo metadata exposes essential fields that I need (targets), but not all (readme, custom package info), so I have to parse both TOML and JSON. Because of that my support for workspaces is half-broken, because cargo metadata supports it, but my parsing of Cargo.toml is naive.

First version of cargo-deb just parsed Cargo.toml, and it was a mistake. It’s deceptively easy to get it 80% right, but impossible to get it 100% right without duplicating (and risking being out of sync with) cargo’s internal logic for interpreting the data and combining it across workspace.

So what I need is full access to all workspace-wide Cargo metadata derived from all the TOML files. It could be a JSON dump or a programmatic API (programmatic API preferred, since parsed JSON makes a de-facto API, but not a very nice one).

1 Like
7

Have you filed an issue for this? It sounds like an excellent E-good-first-bug to me :slight_smile:

programmatic API preferred, since parsed JSON makes a de-facto API, but not a very nice one

There's a semi-official crates.io: Rust Package Registry crate which turns JSON into API. Does it suit your needs? Or do you want "cargo as a library" for programmatic API? The letter has a drawback that the version of Cargo you link with might differ from the user's version.

8

I will think of this

  • a special crates for cargo and build things, so other people could go there and know about such other cargo/build specific things.
  • following the above rustup and cargo should probably refer to that repository of things related to specific extensions for any of them.
  • a possible verbose output that can be mapped to rustc commands? so it would be easy to fine tune things out of this?
9

Not sure if this belongs here: It would be nice if the rust application/library being built could query its own cargo.toml file (at compile time). If this were possible you could have a single source of truth for the information in cargo.toml (things like version number, application description etc.)

e.g. let version = from_manifest!("version");

10

At least some of that information is exported as environment variables, so your example works today as:

let version = env!("CARGO_PKG_VERSION");
11

Thanks, I did not know that! However I would still prefer having more direct access. Environment variables can be unreliable as they are susceptible to interception (think MITM but less intentional more accidental). Of-course some may view this as a “feature” instead of a bug :slight_smile:

1 Like
12

env! grabs the environment variable at build time, and it gets embedded in the binary as a string; it won’t call getenv at runtime.

13

Yes, I understand but in my experience build systems reliant on environment variables tend to be unreliable and lead to hard to debug problems down the line. Though admittedly it is not going to be a big problem here.

1 Like
14

How else would you suggest that cargo provide the program and its build script with a stack of key-value pairs at build time, if not an environment variable? I don’t see any obvious means by which the environment would work any worse than any other such mechanism.

15

I can’t think of a good reason why I wouldn’t use environment variables here either. Probably because I have wasted a lot of time debugging env-var related issues in the past. Now that I think about it more I feel all of those issues were due to “human error” and do not have any bearing on the topic at hand (since the env-vars are set programmatically and stale values do not stick around). In short we should be fine.

16

There should be a programmatic API, so that build scripts don’t need to depend on how the data is passed:

extern crate cargo_build;

fn main() {
   cargo_build::pkg_version();
}

and then cargo can use env vars, or command line arguments, or sockets, or smoke signals.

For CARGO_* env vars that’s not a big deal, but for custom options that build scripts read (e.g. LIBFOO_STATIC=1) it’s a problem, because the vars are invisible to Cargo, and Cargo fails to rebuild projects properly when the 3rd party vars change.

1 Like
17

This is literally why cargo added rerun-if-env-changed=VAR so build scripts can indicate which env vars they use so Cargo can rerun the build script as needed.

1 Like
18

Oh that’s useful. But now it will be an endless whack-a-mole to find all the build scripts that forget to set it. If Cargo had a proper Rust API, then this could be set automatically when a build script reads a setting.

19

That still won't help cases where you shell out to some program (e.g. pkg-config) that reads env vars from the environment.

20

Good point, fixed! https://github.com/alexcrichton/pkg-config-rs/pull/45/files (but as you see in the PR, having a helper for read-and-mark is useful, too)