The burden of creating new compiler options and exposing them in Cargo

compiler
1

There's quite a lot of manual boilerplate in rustc and cargo required to support every option.

In rustc (using this as an example):

  1. It needs a sym defined for the option name, and each of its values, and create a function listing all valid symbols.
  2. It needs to define an internal enum or struct for its parsed value
  3. It needs to define a parser converting from CLI string flag to the internal value
  4. It needs error text added for the CLI parser, because the parser returns false instead of Err
  5. It needs help text added to the list of CLI flags
  6. It needs a cfg(option) defined
  7. It needs UI tests for the new cfg(option) and each of its values
  8. It needs to add the cfg to the list of disallowed user cfgs
  9. It needs to add the cfg to the test of the list of disallowed cfg
  10. It needs to bless the output of check-cfg tests and a few others
  11. It needs to add the cfg to the list of allowed cfgs in CheckCfg
  12. It needs check-cfg documentation updated
  13. It needs a help chapter in the unstable book
  14. It needs to be added to the list of unstable flags
  15. It needs to be added to the default session configuration, gated on nightly options
  16. It needs to be added to the test of options tracked by the session
  17. It may need #![feature(option)] added to tests and libraries

Then in Cargo:

  1. It needs to be added to util::toml structs for the appropriate Cargo.toml section
  2. It usually needs a custom struct with a TryFrom or serde::Deserialize implementation to parse the value from multiple TOML representations.
  3. It needs to be added to the list of nightly Cargo features
  4. It needs help text and parser added for the nightly Cargo feature flags
  5. It needs nightly-only tests checking the TOML parsing and the -Z flag
  6. It may need additional logic and tests for passing the option down to rustdoc
  7. It needs to be added to internal structs representing options
  8. It needs to be added to the hash tracking build options
  9. It needs to be added to code serializing rustc flags, and update countless Cargo text output tests that expose the flags
  10. It needs to be added to functions constructing default internal options and functions converting from TOML-deserialized structs
  11. It may need to be added to config.toml representation, and have code added for merging of options, along with tests for parsing TOML as well as test automagic mapping from env vars to config.toml options
  12. It may need to be exposed as an env var in build scripts, along with bespoke tests for this
  13. It needs to be added to Cargo's option reference.

and then when stabilizing:

  1. The nightly-only option gates need to be deleted.
  2. The lists of unstable options need to be updated to make them stable, separately for rustc and Cargo.
  3. Tests need to be updated to test the stable version of the CLI flags.
  4. Cargo output tests need to be updated to allow the stable flag.

That's A LOT, and this doesn't include any of the actual functionality behind the flag! These steps are just for making a flag exist, and get passed between Cargo and rustc.

A few steps like definition of the flag, its syntax, and its documentation are obviously necessary. But a lot of steps around parsing, serializing, and testing of the parsers and serializers seems to needlessly require bespoke code and bespoke tests.

3 Likes
2

Do all compiler options need to be mapped to cargo though? I saw some recent discussion about adding rustflags support to cargo (just for bin-crates I think), which would make part of this pass-through.

3

Not all flags, there certainly are some low-level plumbing ones that aren't useful for Cargo or don't map 1:1 to something in a manifest.

However, I think it's common that if some new functionality or configurability is added to Rust in rustc, that Cargo users would also want to be able to use it (e.g. fmt-debug and location-detail should be in Cargo profiles). For majority of Rust users Cargo is the primary way they build the code, so Cargo manifests and config are the right place to set the options.

The current RUSTFLAGS env var is a pretty bad interface. You can set only one list of flags for all invocations of the compiler, for both build-time and final binaries, and for both the host and (cross) compilation target. Any change to flags invalidates the build cache, even when the flags have a limited effect. This is coarse and fragile.

I think existence of profile.rustflags is a symptom of the problem I'm highlighting here. It's just so hard and tedious to add individual options to Cargo.toml sections like [profile], that it's easier to give up on TOML, give up on using structured data types, give up on consistency with the rest of the manifest, give up on smart and selective merging of options, and just shove every existing and future option into a single opaque list of strings, using a flat CLI syntax instead of TOML, inside TOML.

5 Likes
4

If it's about the syntax then we could have some sort of toml or json tree passthrough to rustc as an alternative way of specifying options?

5

RUSTFLAGS are opaque to Cargo which makes it hard to reason about. For this reason, the general feel I have for the Cargo team is we view them as an escape hatch for special cases or when we've not yet come up with what abstraction belongs in Cargo. We even have Tracking Issue: Exposing RUSTFLAGS in cargo · Issue #12739 · rust-lang/cargo · GitHub for finding the appropriate abstraction in Cargo for RUSTFLAGS people use. Note this doesn't mean we'll always do a 1:1 between rustc and cargo.

I believe its 1.85 that will at least stop thrashing the cache on RUSTFLAGS changes but will instead keep a distinct cache per RUSTFLAGS.

If the flags do have a limited effect, then that is an example of one of the considerations Cargo needs to take into account when abstracting RUSTFLAGS. For instance, the flag might only be relevant for linking of the final binary. LTO used to be in this category but we then sped up the builds of all crates by building less when LTO is activated.

3 Likes