Pre-RFC: Rust Code Coverage integration in Cargo

Summary

This RFC aims to improve the process of collecting code coverage data for Rust libraries. By including Cargo in the process of instrumenting Rust libraries and running the unit tests, the sequence of steps to get coverage results will be simplified. This RFC also proposes adding support for cargo to selectivly choose which crates get instrumented for gathering coverage results.

Motivation

Why are we doing this? What use cases does it support? What is the expected outcome?

The motivation behind this feature is to allow for a simple way for a Rust developer to run and obtain code coverage results for a specific set of crates to ensure confidence in code quality and correctness. Currently, in order to get instrumentation based code coverage, a Rust developer would have to either update the RUSTFLAGS environment variable or cargo manifest keys. This would automatically enable instrumentation of all Rust crates within the dependency graph not just the top level crate. Instrumenting all crates including transitive dependencies does not help the developer ensure test coverage for their own crate.

Guide-level explanation

This section examines the features proposed by this RFC:

CLI option

A new subcommand for the cargo test command would be added. The new command --coverage would instruct Cargo to add enable the Rust flag -C instrumental-coverage, for the given crate only. This would mean that only the top-level crate would be instrumented and code coverage results would only run against this crate. As an example, lets take the following crate foo:

/Cargo.toml
  +-- src
      +-- lib.rs

Where crate foo has a dependency on the regex:

[dependencies]
regex = "*"

And lib.rs contains:

use regex::*;

pub fn add(left: usize, right: usize) -> usize {
    left + right
}

pub fn match_regex(re: Regex, text: &str) -> bool {
    let Some(caps) = re.captures(text) else {
        return false
    };

    let matches = caps.iter().filter_map(|s| s).collect::<Vec<Match>>();
    matches.len() > 0
}

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn it_works() {
        let result = add(2, 2);
        assert_eq!(result, 4);
    }

    #[test]
    fn find_match() {
        let result = match_regex(Regex::new(".*").unwrap(), "Hello World");
        assert_eq!(result, true);
    }

    #[test]
    fn find_no_match() {
        let result = match_regex(Regex::new("a+").unwrap(), "Hello World");
        assert_eq!(result, false);
    }
}

Now running cargo test --coverage would produce the coverage results for the foo crate only and ignore all functions defined outside of this crate.

Reference-level explanation

As mentioned earlier this RFC proposes adding a new subcommand to the cargo test command. This subcommand, --coverage would be responsible for setting the -C instrument-coverage flag that Cargo would pass on to the rustc invocation of the top-level crate. In the previous example, foo would be the top level crate and regex would be upstream an dependency.

Using the --coverage subcommand, Cargo would only manually set the -C instrument-coverage flag for the crate foo. If the RUSTFLAGS environment variable had already been set to include the -C instrument-coverage flag, then Cargo would still pass that flag to all crates within the dependency graph, including the regex crate and any transitive dependencies.

This should not break any existing workflows and is strictly an opt-in feature.

To use this new feature do the following:

cargo test --coverage

This subcommand would also be responsible for setting the LLVM_PROFILE_FILE environment variable which is read by LLVM to generate a .profraw file for each test executable that is run. Once again if the environment variable is already set, then Cargo would not make any changes and would leave the value as is to use the user-defined file name. If the environment variable is not set, Cargo would set it to ensure a unique naming scheme is generated for each .profraw file that would be generated.

These updates to Cargo would be sufficient enough to ensure that a Rust developer would have control over what crates are instrumented and code coverage results are generated. This would also allow the Rust developer to no longer have to set environment variables manually to ensure crates are instrumented for gathering coverage data.

Drawbacks

A drawback of this feature would be that Cargo would need to enable the LLVM_PROFILE_FILE environment variable in order to ensure unique profile data is generated for each test executable. I am not aware of any other Cargo features that set environment variables today so this would be a new potential problem introduced for Cargo.

Rationale and alternatives

Rationale

This design provides a simple mechanism to integrate collecting code coverage results for a given crate. Allowing Cargo to be part of the coverage process would reduce need for setting environment variables manually. Simply running cargo test --coverage would automatically run a build setting the -C instrument-coverage Rust flag and set the LLVM_PROFILE_FILE environment variable to ensure each test run produces a unique profraw file.

This design does not break any existing usage of Rust or Cargo. This new feature would be strictly opt-in. A Rust developer would still be able to manually set the -C instrument-coverage Rust flag and instrument all binaries within the dependency graph. Since this is a Cargo specific feature, the Rust compiler will not need any updates.

Alternatives

Alternative 1: leave the existing feature

Supporting this alternative would mean that no changes would be necessary to either Cargo or Rust. Getting instrumentation-based code coverge is still supported and would continue to work as it does today.

The drawback for this option is that it would require setting the flag for all crates in the dependency graph, including upstream dependencies. This would also instrument all binaries and report on coverage for functions and branches that are not defined by the current crate with the potential of skewing coverage results.

Alternative 2: use a new manifest key instead of a cli option

Supporting this alternative would mean that changes would need to be made to the existing manifest format to possibly include a new section and/or key. A new coverage key could be added to the target section, coverage = true. This still has the added benefit of not requiring any changes to the Rust compiler itself and the feature could be scoped to Cargo only.

The drawback for this option is that it could potentially add clutter to the Cargo.toml files. Adding this new section to every crate that a Rust developer wants to have instrumented would only add to all the data that is already contained within a toml file.

Alternative 3: use a RUSTC_WRAPPER program to selectively choose which crates to profile

Supporting this alternative would mean that there wouldn't need to be any changes to Cargo at all.

This would require creating a RUSTC_WRAPPER program specifically for selecting which crates to profile. This means more boiler plate code for each Rust developer that simply wants to profile their own crate. I believe the feature this RFC proposes would both be a cleaner solution long term and more in line with the Cargo workflow of potentially reading these kinds of behaviors from the Cargo.toml manifest file.

Prior art

VSInstr

Visual Studio ships with a tool vsinstr.exe which has support for instrumenting binaries after they have already been built. Since LLVMs instrumentation-based code coverage hooks into each object file it generates this scenario is a bit different than the feature this RFC proposes. vsinstr does allow for excluding namespaces of functions to skip over so that everything within a binary does not get instrumented.

gcov based coverage

Rust also has support for another type of code coverage, a GCC-compatible, gcov-based coverage implementation. This can be enabled through the -Z profile flag This uses debuginfo to derive code coverage. This is slightly different than the source-based code coverage which allows for a more precise instrumentation being done.

Unresolved questions

  • Are there any drawbacks from having Cargo set the LLVM_PROFILE_FILE environment variable that LLVM uses to name each of the generated profraw files? This is used to ensure each test run generates unique profiling data as opposed to overwriting the previous run.

Future possibilities

Specifying multiple crates to instrument for Code Coverage

This would allow for Rust developers to specify each of the crates they want to instrument in advance and Cargo would be able to pass on the -C instrument-coverage flag for only the crates specified. This would allow a more targeted approach of getting code coverage results and not for developers to instrument the entire dependency graph. This could either be in the form of a manifest key in the toml file which would take a , separated list of crates to include in the code coverage analysis or by specifying each crate at the command line using --coverage:crate_name.

1 Like

So, basically this is about merging cargo-llvm-cov into cargo source tree?

2 Likes

Given that cargo-llvm-cov exists, I see no reason to integrate this into cargo proper. Code coverage just landed on stable, so the ecosystem hasn't even had time to develop complete solutions. I think it's far too early to be talking about anything integrated.

6 Likes

Do you imagine this functionality just as a feature of cargo test, or would the plumbing for this involves a --coverage feature on cargo build?

I ask because a project I work on currently leverages -C instrument-coverage to cargo build in order to build Python extension modules with Rust coverage. So we'd benefit from this being deeply integrated.

Notwithstanding that, the ergonomics of coverage now are a bit rough. All the functionality is there, but it's spread out across CLI flags and different binaries. Making coverage feel more native to Rust would be very valuable for the developer experience IMO.

5 Likes

How would this work in conjunction with the --frozen flag?

I disagree. The way I do coverage now is with tarpaulin. I split the build from the test because the test sometimes needs additional permissions (docker run --privileged flags for coverage) or other environmental differences that wouldn't require a fresh build (e.g., using a snapshot of git versus a release version). Only being able to access the coverage build from the test command would be disadvantageous as it would expose the build to elevated permissions or access in CI.

I recommend adding tarpaulin to this.

I think even exposing that the envvar does anything would tie down the implementation too much. Any user control that might be available should be performed through RUST_ envvars or cargo flags.

1 Like

I was not aware of cargo-llvm-cov, thank you for the link to this. It definitely seems like what I am trying to accomplish here. I'll look more deeply into this tool.

Thanks for the response!

How would this work in conjunction with the --frozen flag?

It should still work as it does today. Adding the instrument-coverage flag shouldn't have an effect on the lock file which is basically the main thing this feature would be doing.

I disagree. The way I do coverage now is with tarpaulin. I split the build from the test because the test sometimes needs additional permissions (docker run --privileged flags for coverage) or other environmental differences that wouldn't require a fresh build (e.g., using a snapshot of git versus a release version). Only being able to access the coverage build from the test command would be disadvantageous as it would expose the build to elevated permissions or access in CI.

Thank you for pointing me to tarpaulin. This definitely seems like another tool that deserves to go into the prior art section. I'll do a bit more digging into this as well as cargo-llvm-cov to see the similarities and differences in behavior.

I think even exposing that the envvar does anything would tie down the implementation too much. Any user control that might be available should be performed through RUST_ envvars or cargo flags.

This user control exists today solely due to how LLVM creates the profraw files. With this feature we would not want to break existing users who are depending on a specific naming scheme.

Do you imagine this functionality just as a feature of cargo test , or would the plumbing for this involves a --coverage feature on cargo build ?

This is a great question. This feature could potentially go in that direction but for the current scope of the RFC it was meant solely for the use case of getting coverage data for Rust tests. I believe this might work best in the Future Goals section of the RFC to have a bit more discussion around this.

That is the --locked flag. --frozen additionally requires that any pre-requisite have been performed (e.g., downloading deps for build and test --no-run for test). I suppose cargo test --no-run --coverage just needs to be tested here.

As other have mentioned, cargo-llvm-cov works quite well and really does a great job of exposing -Cinstrument-coverage in user-friendly way! It would be great to have support for instrumenting and colling coverage data inside cargo itself but, if we do that, I think the end goal should be to make it even easier, perhaps even enabled by default, to expose this information to users. I can imagine cargo test outputing something like:

running 4 tests
test foo::tests::test1... ok
test foo::bar::tests::test2... ok
test foo::bar::tests::test3... ok
test foo::baz::tests::test4... ok

test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s
coverage: crate `foo`: 53% covered; crate `bar` 81% covered

While I think this would be a great quality-of-life feature, it also seems like a lot of work...


Currently, in order to get instrumentation based code coverage, a Rust developer would have to either update the RUSTFLAGS environment variable or cargo manifest keys. This would automatically enable instrumentation of all Rust crates within the dependency graph not just the top level crate. Instrumenting all crates including transitive dependencies does not help the developer ensure test coverage for their own crate.

I think this really hints at a missing feature that I've occasionally wanted: the ability to pass rustc flags for just "my" code. Code coverage is a good eample; I don't particularly care how much of my dependencies code my tests cover so what I want is to be able to have cargo only pass -Cinstrument-coverage when it's building my code. For a single crate, this is easy, just use cargo rustc. For a workspace of crates, I'm not aware of an easy way to do this currently.

Similarly, I might not need detailed debuginfo for dependencies, but I would like it for my crates. I believe this can be done today using profiles and the debug manifest key.

Perhaps this means we should either focus on adding a new manifest key for code coverage and/or we should explore some kind of "rustflags" manifest key that can serve as a more general solution to this problem? I guess there's also potential for some kind of RUSTFLAGS_JUST_MY_CODE="..." environment variable?

2 Likes

Questions I have that I would like to see acknowledged in an RFC on this:

  • What is the workspace workflow?
  • What about doc tests?
    • It looks like cargo-llvm-cov has experimental support for this. I'm not familiar with the challenges they are running into. I wonder if it'd be easier being directly integrated into cargo (ie a potential benefit of this RFC) or if its just the same
  • How do UI tests integrate with this? Many CLIs do end-to-end / ui tests of the underlying binary being built. See also Coverage reporting missing for binaries under-test · Issue #9 · assert-rs/assert_cmd · GitHub
  • With touching LLVM_PROFILE_FILE, how does this work with alternative backends? (maybe the -C instrument-coverage RFC answers this; I haven't looked)

@wesleywiser Thanks for the comment!

I agree but I believe the end goal would be a seamless way of collecting coverage data for tests that simply doesn't exist today. With tools such as cargo-llvm-cov, it relies on unstable cargo commands such as cargo-config.

I also believe that having the integration built into Cargo specifically will remove the need for any unstable features such as cargo-config since access to the Cargo configuration and toml files will be readily available. This would also help the Rust community collect coverage results automatically from unit tests via cargo without having to install other tools. Whether it is on by default, a subcommand for cargo test, or a new subcommand for Cargo directly such as cargo coverage.

Yes this seems like a feature that is missing, but this would stop short of a great experience for code coverage since it will only add the ability set the Rust flag and not actually expose the coverage information to users running tests. I believe a feature integrating coverage support into cargo which is typically the way in which Rust developers run unit tests is the more complete feature here.

@epage thank you for taking the time to comment on this and raising some important questions!

In terms of a cargo workspace, I believe there should be an option to specify a package, --package, to instrument only specific crates or a --workspace option to allow cargo to use the set of workspace members defined as the set of crates which would be instrumented.

This is currently out of scope at the moment for this RFC, but I could definitely see this being added as part of the Future Goals section of this RFC.

In terms of integration tests, once the Rust libraries have been instrumented, coverage information is collected anytime an executable is run that links to these libraries such as the executable created for a Rust integration test. If you are speaking of end-to-end tests which use a different test runner than cargo, those specific cases are out of scope for this RFC.

I believe the -C instrument-coverage is only supported for the LLVM based backend currently and the LLVM_PROFILE_FILE environment I believe is only read by LLVM.

1 Like