Improving support for coverage-guided fuzzing

Hi. I am the author of fuzzcheck, a coverage-guided fuzzer written in Rust.

One of the things that fuzzcheck needs to work properly are hooks in the user’s code that are called whenever a new code block is reached (i.e. the code must be “instrumented”). In practice, I use LLVM’s SanitizerCoverage for that purpose, just like cargo-fuzz.

To be able to use SanitizerCoverage, the code must be compiled with the following options:

-Cpasses=sancov
-Cllvm-args=-santizer-coverage-level=N    (N = 1, 2, or 3)
-Cllvm-args=-sanitizer-coverage-trace-pc-guard

There are also other useful options that insert more/different hooks, but the ones above are the basic ones.

Then, in the fuzzcheck crate, I implement the hooks, for example:

#[export_name = "__sanitizer_cov_trace_pc_guard_init"]
unsafe fn pcguard_init(start: *mut u32, stop: *mut u32) {
	// save the [start, stop) array in a global mutable variable
}
#[export_name = "__sanitizer_cov_trace_pc_guard"]
unsafe fn trace_pc_guard(guard: *mut u32) {
    *guard += 1; // for example
}

It is fairly simple and it works. But a big problem I have is that the compilation options shown above apply to the whole crate graph, and I have not found an easy way to selectively instrument some crates and not others. It is absolutely essential that the fuzzcheck crate itself is not instrumented. If it was, then the hooks would trigger themselves, causing infinite recursion.

To get around this problem, fuzzcheck comes with a tool called cargo-fuzzcheck. Its main purpose is to compile the user’s code and the fuzz targets correctly. First, it creates an instrumented crate that contains the user’s code and the tests. It compiles that crate using the SanitizerCoverage flags. Then, it creates a non-instrumented crate that contains fuzzcheck and its dependencies. It manually links the instrumented crate into non_instrumented.

But it is very hack-y and confusing. Furthermore, I still end up instrumenting code that I don’t want instrumented, which results in a big loss of performance. For example, the fuzzcheck_mutators crate that is needed to perform structure-aware fuzzing is usually instrumented because it may need to know about private implementation details of a user’s type. I believe that cargo-fuzz also suffers from this problem, with the arbitrary code being instrumented (I even think its coverage is tracked and informs the fuzzer’s decisions, which is not good either). And finally, it is difficult to ensure that some common dependencies of both non-instrumented and instrumented are not compiled twice.

What I’d like to provide, eventually, is an interface that is very similar to today’s unit or integration tests. For example, something like:


#[derive(fuzzcheck_mutators::DefaultMutator)]
struct MyType {
   // ...
}
impl MyType {
    fn do_stuff(&self) -> Y {
        // ... 
    }
}

#[fuzz_test]
fn my_fuzz_test() {
    // this could be simpler of course, but maybe we can think of adding more magic later
    let mutator = MyType::default_mutator();
    let serializer = JsonSerializer::default();
    let test = |x: &MyType| {
        let y = x.do_stuff();
        y.is_correct()  
    };
    fuzzcheck::launch(test, mutator, serializer);
}

Then one could run my_fuzz_test using either cargo or, still, a third party tool like cargo-fuzzcheck. But the point is that the fuzz test would not be its own executable residing in a very carefully managed fuzz folder and needing a complex procedure to compile.

If I missed something that could accommodate fuzzcheck’s use case, please let me know. I have tried to look at current efforts that could improve the situation and have found a couple things:

  • There is already a (nightly-only) attribute that can selectively disable some sanitisers for annotated functions (link), but it does not support SanitizerCoverage. How difficult would it be to add a coverage option to it? Could it apply to an entire crate?
  • Maybe I could also move away from LLVM’s SanitizerCoverage and use Rust’s -instrument-coverage option (link) instead? The problem is that it comes with its own profiler runtime. It doesn’t look easy to write a regular crate that provides a different profiler runtime for it, or to customise what code is instrumented. I am also not sure of its impact on code speed. But it certainly looks attractive to use a tool that (1) is fully supported by the Rust compiler and (2) can also provide detailed coverage information.

I don’t follow Rust’s development very well, and I have never worked on rustc or cargo. But I am willing to learn the codebase and spend time implementing the features that are necessary to provide better support for coverage-guided fuzzers written in Rust.

I’d like to talk about:

  • whether this is a goal that the rest of the Rust community think is worth pursuing
  • whether the #[no_sanitize] attribute could be extended to support SanitizerCoverage
  • the design of -instrument-coverage and its ability to serve as the instrumentation tool for coverage-guided fuzzers

Thank you!

2 Likes

You might want to dig into cargo-tarpaulin to see how it instruments code. I believe it only instruments the current workspace's crates, but I'm not sure how it does that. Of course, it could be instrumenting everything and just ignoring results from outside the workspace too, but it might have some helpful filtering logic in there if it is doing something smarter.

Try taking at look at bolero and see if you can use any techniques from there. I don't think it's a feedback fuzzer, but it may have some techniques that can help.