Tools Team: tell us your sorrows

[spacekookie]

As part of the new and improved Tools team, we want to hear from users and tool-makers alike about things that you find lacking or suboptimal in the Rust tooling space.

This could either be things that tools exist for, but aren’t yet fully solved, or where tools are missing entirely.

Depending on the feedback to this post we might decide to also run a more official survey, focusing on details about what can be improved.

Giving feedback

In order to make the responses easier to parse, analyse, and structure, please try to stick to a common form in your feedback.

• What area of tooling is affected?
• Is there a tool for this already?
  • If so: which one?
  • If not: did you write one?
  • If not: which tool do you think it should be included in?
• Did you manage to work around the lack of tooling?

Edit as a small reminder though, this shouldn’t include all feedback about all tooling that is generic to Rust (like the compiler). There’s always gonna be things that can be improved on rustc (i.e. compile times) that aren’t immediately relevant for the Tools Team!

Notes (edits)

• Debugging: please check the previous comments if your points have already been raised
• rustup: producing nightly versions that contain all tools reliably

[Shnatsel]

Area: Unsafe code verification, dynamic analysis.

Detecting reads from uninitialized memory is really hard. This is especially unfortunate because they may result in a security vulnerability - both information disclosure and arbitrary code execution.

Existing tools: Memory Sanitizer, libdiffuzz, Valgrind, miri

Memory Sanitizer exists and kinda sorta works with Rust, but it requires you to recompile libstd (among other things) and the process is not documented. Once you get over the “recompile libstd” step, there are other hurdles. Using it was so hard that I’ve literally decided writing a custom tool would be easier.

The custom tool I’ve written is libdiffuzz. Sadly, it is not very useful for debugging and is not 100% reliable - it may miss bugs.

Valgrind is not usable for me due to the inability to combine it with fuzzing. I also personally dislike it for the massive slowdown it introduces and high rate of false positives. People who have tried it have reported that it outputs nonsensical results. This was before the switch to system allocator on Linux, though. AFAIK there are no up-to-date docs.

MIRI is impractically slow for most real-world code. I have tried to verify if decoding a 5Kb GZIP file with a Rust implementation if DEFLATE results in reads from uninitialized memory; after an hour the process was still not completed.

[Shnatsel]

Area: Unsafe code verification, dynamic analysis.

Address Sanitizer is great, but Rust support is not in a great shape right now. It requires obscure workarounds to get going.

https://github.com/rust-lang/rust/issues/53945 impacts all sanitizers. This can be worked around, but the workaround is obscure and introduces an additional hurdle. Probably platform-specific as well (I am only aware of a Linux option).

Also, you need to include the following in the executable, otherwise you will face false positives on some programs:

// suppress ASAN false positives
const ASAN_DEFAULT_OPTIONS: &'static [u8] = b"detect_odr_violation=1\0";
#[no_mangle]
pub extern "C" fn __asan_default_options() -> *const u8 {
    ASAN_DEFAULT_OPTIONS as *const [u8] as *const u8
}

I can live with that, but it dramatically raises the entry barrier, which is a shame for such an essential tool.

There are some alternatives to ASAN, such as libdislocator, DUMA and some modes of Valgrind, but ASAN is by far the fastest and most reliable of the bunch.

[Shnatsel]

Area: security updates

When running Rust binaries, you do not get automatic security updates. Rust binaries are statically linked, so you cannot not get a shared library update through the usual channels to mitigate the exploit. Nothing will inform you about the exploit or nag you to update. There is no way to even tell if you’re running a Rust binary with known security vulnerabilities because the list of libraries that went into creation of a binary is not preserved.

RustSec maintains a Rust security vulnerability database, so machine-readable information about known vulnerabilities exists.

https://github.com/RustSec/cargo-audit can inspect your Cargo.lock for known vulnerable dependency versions. The catch is that you need to run it manually, and nobody has the time to run it on checked out versions of their source code every day. Not to mention that the source code does not necessarily correspond to the binaries deployed in production.

https://github.com/Shnatsel/rust-audit is a very much proof-of-concept tool written by me. It embeds Cargo.lock into the compiled binary and provides a tool to extract it; it can be then fed to cargo-audit to audit the binary for vulnerable dependencies. It is not ready to use as-is, and for this concept to work this needs to be enabled by default in all Cargo builds except WASM and embedded platforms.

https://gitlab.com/zachreizner/crates-audit/ inspects crates.io for crates with dependencies without a semver-compatible upgrade path to mitigate the exploit. Pretty much nobody has heard of it; crates.io does not report presence of known vulnerabilities in dependencies. It has configurable output format; html output of a recent run can be found at https://crates.rustsec.org/

Other pieces of the puzzle such as regularly scanning binaries pulled via cargo install for vulnerable versions, alerting the user to the vulnerabilities or automatically installing security updates that people have come to expect on Linux for the past 20 years are missing entirely.

This will become even more critical once async/await lands and people start building lots of network-facing programs in Rust.

[kornel]

Area: Cargo packaging/project management

Cargo.toml can’t contain all project settings, and some flags have to be put in .cargo hidden directory. It’s PITA to have part of the config in a separate hidden file. https://github.com/rust-lang/cargo/issues/1137

Worse, for dependencies (sys crates specifically) there’s no place to put the config at all. They tend to use arbitrary environmental variables, which Cargo doesn’t understand, and can’t manage. And there’s annoying misfeature that prevents tools like pkg-config from supporting env vars better: https://github.com/rust-lang/cargo/issues/4587

I’d really like Cargo.toml to be able to fully describe my project, so that when I run cargo build it builds it correctly. Not fiddle with rustflags in the hidden config, and not fiddle with env vars that have to be set from another build system wrapping Cargo.

I’m worried that Rust packaging projects approach the problem from wrong angle - by blindly packaging whatever Cargo has (incorrectly) built, because there’s no correct way to configure project for different packaging targets.

[kornel]

Area: Rust in Linux distributions, handling of incompatibilities

Users file bugs against my crate that “it doesn’t compile”, “can’t parse Cargo.toml” or “wants Nightly rust”, while the real reason is they have an outdated Rust version.

Cargo doesn’t have a way to warn users about not meeting required Rust version.

Currently I tell people not to use Rust from Linux distros and instruct to run rustup update before build.

[ssokolow]

A good stop-gap for that sort of thing that shouldn’t require a huge RFC process might be to amend those error messages so that, if the toolchain version is more than six weeks old, it adds something like this to the end of the message:

Upgrading Cargo and rustc may solve this problem.

[whentze]

Area: Tools that need to analyze rust code.

Being able to analyze arbitrary rust code is a need for quite a few tools. However, if a tool needs more in-depth analysis than just parsing, it often needs to depend on rustc internals, which are completely unstable.

This issue concerns tools shipped via rustup (rls, clippy and rustfmt) but also third party crates (rust-semverver, mutagen etc.). The former are sometimes missing in nightly due to breakage and the latter will also often fail to compile.

I’m not sure in general what the way forward is here, but I would like tool builders to have a nightly-free and future-proof way to analyze rust code. There are two approaches that already work for some existing tools and may be worth expanding upon:

1. Tools that just need parsing can use the great syn crate. It is a regular stable rust crate and can be used like any other dependency. I know that features above the syntax level change much more frequently, but in the future, as Rust matures and (hopefully) stops being such a moving target, stable crates for higher-level analysis should be a goal.

2. Some tools instead opt to consume the machine-readable output already produced by rustc. This works great for tools like rustfix that can use information that is very similar to what a human user of rustc sees. This approach has the benefit of allowing tooling that is not written in Rust, as the interface is completely portable. The downside is that it limits you to what rustc is willing to emit in this way, which is currently quite limited.

I’m really not sure whether either of these is the way to go, or if there’s another one I’ve overlooked, but you asked for sorrows so here they are.

[notriddle]

Area: Windows packaging.

The already-existing tool for producing Windows MSI installers, cargo-wix, is maintained by one guy. It shows. In order to include an entire directory in my WiX installer, I need to produce a second XML file and pass it along with the first to the candle CLI tool. cargo-wix has no support for this. I’m currently using a hacky fork; the code is not appropriate to upstream.

I also need to give my binary a proper embedded icon, and I need to have a manifest, but neither cargo-wix nor the bare Rust distribution itself support that at all. I am currently working around this using the RUSTFLAGS env variable and a batch file that wraps link. This only really works correctly from the shell scripts that wrap this whole thing in CI: developers working on their local machines can’t just run cargo build and expect it to work (they get obscure internal Windows errors because the binary can’t locate its tray icon).

[wildstyle]

detect_odr_violation=1

BTW has this been reported to google as a defect in ASan’s design? It’s unnecessarily bound to a C++ language specification. Presumably they want it to be portable and useful to more than just C/C++ developers. At a minimum, offering a way to descope the build such that it’s not available (or even off by default) would suffice.

[wildstyle]

while the real reason is they have an outdated Rust version.

I think this will converge to rustc and cargo acknowledging the edition, right? The problem is just that older rustc’s (relative to now) just don’t look for it because the feature didn’t exist yet.

[canndrew]

Area: rustup

rustup update has been broken for me for ages because I use nightly and rls. I used to be someone who was always up-to-date but now my nightly is months behind because it’s a PITA to find a nightly which has rls and switch to that specific nightly. nightly should either always include all components or there should be some kind of nightly-complete branch which tracks the latest nightly which includes all components.

[jgarvin]

As a new user the most immediate frustration with the tooling was the lack of a single straightforward place to consult for the most up-to-date series of instructions for getting my IDE of choice working with all of the bells and whistles made possible by the language server (emacs). There were some resources talking about needing “racer”, some talking about needing RLS, and almost everything was telling me to install nightly even though nightly is no longer required. Once I got over that hurdle the figuring out that RLS is what I wanted, I still had to consult obscure Reddit threads to find workarounds for obvious bugs and problems. Also at one point RLS was a separate emacs package but now is included in the main language server package, but all the guides were still telling me to install the old package. Then it turned out there are two different fly check modes, one that uses RLS and one that doesn’t, but their functionality doesn’t totally overlap so you want both, which is doubly confusing. It would be nice if there was an agreed upon place for guides for each IDE or editor that would have a maintainer that would make sure that the instructions are up to date for the current stable version of rust.

The second most frustrating aspect was the tooling did not consistently work – for example the language server could find definitions for my code but not from the standard library (not even println!) or third party crates.

[happycoder97]

Area: Intellisense for code generated by macros

It is difficult to work with libraries like Diesel, which uses lots of macros for code generation, due to the lack of intellisense. Diesel is a library that we have to use frequently in backend projects.

There is no tool for this already. I couldn’t manage to work around the lack of tooling.

[crlf0710]

Area: Debugging

Need C/C++ alike debugging facility and quality, proper integration with an IDE like (but not have to be) Visual Studio, properly displaying basic structures like enums and slices, step-into/step-over/run-till-end-of-function, breakpoint and breakpoint counter. Proper handling variables with same name but one shadowing another.

Currently a show-stopper for corporate usage at my site.

[orthoxerox]

RLS gives valuable hints about missing items, but editor plugins (or at least the VS Code one) do not currently use this information to generate and insert item stubs. Trait implementation is the biggest pain area, but being able to generate function stubs from invocations would be very convenient, too.

[mindtree]

Area: Compilation Time

We often run workshops for our creative coding framework nannou. Hands down the biggest issue we face with these workshops and the most consistent feedback we get from attendees is compilation time. There are a couple specific areas:

• Initial Compilation - The first compile of nannou can take up to 20 minutes on some users machines, which is a big roadblock during the workshop when we would like to be spending that time focusing on the examples and showing users how they can be creative. A potential solution for this might be to allow for specifying pre-built packages as dependencies, where cargo falls back to a builid from src if the pre-built source doesn’t contain a binary with a compatible architecture.

• Caching Build Artefacts Between Crates - After we work through the examples with the user, we like to show users how they can create their own sketches in their own crates. Currently however this means the entirety of nannou must be built again, this time as a dependency to their sketch, rather than building the package itself, resulting in another potential 20 minute build time for users with slower machines. This is particularly frustrating when we know all of nannou is already built, just in a different directory. sscache shows potential for solving this, however it is currently quite unreliable and requires installing another tool during the workshops which adds further complexity to the process. We would love to see more focus put on sscache’s interop with cargo and ultimately would love to see the tool integrated into cargo itself. We frequently start new projects, sketches, installations, artworks, festival visuals, etc that depend on nannou and have collectively easily lost days worth of dev time waiting for nannou to build from scratch each time. We often create a workspace and just create new projects within it to work around the issue, but this feels like a hack particularly when all of the projects are unrelated to one another.

[nox]

rustup update already skips new builds for which one of the components you installed is missing.

[nox]

RLS is unusable (I’ve yet to see it start once) in the Servo project, which is quite unfortunate given it is the oldest and one of the largest non-rustc codebases in existence.

[lzutao]

I think what you want to ask is something like rustup install nightly-complete