Disabling --gc-sections when --test is specified?

[alexcrichton]

So right now one of the best ways to get code coverage analysis with Rust is to use cargo test and then use kcov to run the binaries, as outlined in @lifthrasiir’s excellent tutorial on the topic. One of the problems with this approach, however, is how agnostic kcov is to the language being profiled.

It appears that kcov works by looking at all code that’s being run, and then it tracks what’s actually run and what’s not. All not-run code is considered not covered. This generally sounds like a good idea, except that the compiler’s default behavior ends up messing with the output of kcov.

By default the compiler will compile each function and each global into its own section of the output object file. The linker is then passed --gc-sections which will eliminate all unused functions/globals at link time. This behavior is crucial to keep binary sizes under control in Rust, and I don’t think that we should turn this off by default. This does, however, have the implication that testable functions in a library are stripped by default in test executables if they are not used (or in this case, not tested). This consequence of this is that all untested code is not considered for coverage by kcov. Now seeing how one of the great points of kcov is to tell you what’s not being covered, this seems bad!

What would others think about not passing --gc-sections by default when the compiler is passed the --test flag. The consequence of this is that test binaries will be larger, but the benefit is that tools like kcov (I can imagine there might before) will be able to understand what code actually isn’t run and provide more accurate coverage reports. An alternative here would be to add a separate compiler option to not pass --gc-sections, but this would be somewhat more complicated as Cargo would then need to perhaps provide a subcommand like cargo coverage to pass this flag.

Curious what others think!

[wycats]

An alternative here would be to add a separate compiler option to not pass --gc-sections, but this would be somewhat more complicated as Cargo would then need to perhaps provide a subcommand like cargo coverage to pass this flag.

I like the idea of an explicit cargo coverage

[alexcrichton]

My only reservation about cargo coverage is that we don’t have a cross-platform solution for code coverage yet. To my knowledge kcov is linux-specific and I don’t know of any tools for OSX or Windows that do this kind of code coverge.

[vadimcn]

What about measuring code coverage in scenario testing? (i.e. normal compilation, without --test, then running executable on some test data).

[nikomatsakis]

@alexcrichton stupid question, but does this imply that kcov will complain about code in libstd that is not used or anything crazy like that? I guess perhaps it would if things were statically linked, but not when things are dynamically linked? (which makes sense)

[huon]

It shouldn’t, or at least, I don’t see any data from std in the output of kcov now (i.e. not even listed as covered). Although, that may just be because the distributed std has no DWARF debug info in it (I have no idea).

[alexcrichton]

@huon is right here I believe, if we shipped libstd with debuginfo then kcov would by default tell you about code coverage in the standard library, but we don’t do that. kcov, however, has filters to allow you to refine the output, so it’s not necessarily the end of the world!

[ehiggs]

I like the idea of cargo coverage. Why not use LLVM’s baked in code coverage tools?

[mbrt]

Shouldn’t we provide an additional option in the [profile.*] sections of Cargo.toml? Maybe it is simpler than having a whole new cargo subcommand.

[chamakits]

Hi everyone. Until this get solved, I came up with a practical, but likely not elegant work around. The following macro will force your functions to be compiled as part of your test process. Truly not ideal, but gets the job done:

macro_rules! assign_do_nothing {
    ( $( $x:expr ),* ) => {
        {
            $(
                let _curr = $x;
                )*

        }
    };
}

#[test]
fn mock_forcing_functions_compiled() {
    assign_do_nothing![
        super::first_function, 
        super::second_function, 
        super::third_function, 
        super::fourth_function];
}

Is there a way to iterate through functions with a macro, so that a similar concept could be used but without having to explicitly specify the functions?

[mzabaluev]

How about adding a Cargo profile for coverage, which would differ from the test profile by passing this linker option and maybe LLVM flags to assist coverage generation (clang has --coverage, I dunno if this percolates to the LLVM backend)?

[huon]

If -C no-gc-sections (or similar) were added to rustc, this “fancy” switch could be implemented in cargo test (and the choice of --gc-sections or not could be exposed as another profile option, with true as the default for everything except test).

[euclio]

Also very interested in this feature. Our coverage reports in http://github.com/Hoverbear/rust-rosetta are very inaccurate because of this optimization.

[briansmith]

Please do not do anything that makes the settings for cargo test diverge from the settings of cargo build beyond what is currently done. Testing what we will actually ship, as closely as possible, is vital to ensuring the testing is valid.

I like the idea of cargo coverage much more.

[huon]

Testing what we will actually ship, as closely as possible, is vital to ensuring the testing is valid.

One is generally shipping the result of cargo build --release, so making sure cargo test and cargo build match is somewhat of a red herring. We already support cargo test --release which, I believe, uses the release profile rather than the test one.

[briansmith]

We already support cargo test --release which, I believe, uses the release profile rather than the test one.

As long as cargo test --release is built with --gc-sections when cargo build --release is, that’s fine by me.

[alexcrichton]

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.