Pre-RFC: Mutually-excusive, global features

tools and infrastructure cargo
21

The default is specified inside of the manifest and will be set by cargo, so global::sys will always be available in a package that declares it. For packages that don't declare a global, it won't be set and we'll be using the cfg validation feature to turn it into a compile error to reference it.

1 Like
22

I like the idea of this, as there's a lot of times (async backends, server runtimes, embedded memory uses) where the features are manually mutually-exclusive and they should be marked as such instead of misusing cargo features. I'm not too sure about the difference between globals and parameterized packages - is there a non-syntactic difference?

1 Like
23

Globals are controlled by the root crate (e.g. a [[bin]])

Parameterized packages would be controlled by the direct dependent and can cause multiple copies of the package to be built if there is another instance with a different feature set selected. Think generics where the generic parameters can cause multiple copies of a function to exist.

24

As one of the maintainers of pgrx, a crate for building Postgres extensions, which depends on binding against the correct major version of Postgres, and has to customize its bindings for each major version, I'm fairly interested in this. Right now we have the obvious pile of hacks you might imagine to make this work, and what we do is we have a cargo pgrx new that sets our users up with the right template to expose the crate's version to whoever might be depending on it, so that the final user can pick the right version via setting pg15 on the topmost pgrx crate or whatever.

Because of the nature of Postgres extensions, each being basically its own dylib, we don't have to worry much about nested composition. However there are people who have tried to build extension crates which offer helpers for working with our code in an opinionated way, and it would be nice if this was easier for them since they need to update their own featureset every time we ship a new Postgres major version and retire an old one, which happens every year.

25

An interesting potential use case for this is to allow multiple copies of a links to exist in the dependency tree so long as they are coming from mutually exclusive features.

See Resolver is too strict in its multiple-crates-`links`-ing check · Issue #5969 · rust-lang/cargo · GitHub

26

I came here because I was looking whether there's a way to exclude dependencies when a feature is enabled (i.e. use std Mutex instead of downloading spin crate when std feature is on). I know this doesn't tackle dependencies directly but it interacts with them.

Using enum globals is better than on/off, but there are (although rare) cases where other value types might be a better fit. Enums satisfy my use case but I'd like to note some of the issues I see with the current proposal:

  • If I create a heterogeneous_tuple crate which defines a trait for dealing with tuples as a list of types, (due to lack of support for vararg generics) I'd need to generate code for 0..N sized tuples and like to have N be configurable because generic code might require 20 to 60 different types (e.g. strongly typed I/O pins on a circuit simulator). Enum globals don't handle that case well, a numeric global would allow the user to set the value to a required maximum which might be less or much more than anticipated preset values.

    • In the same line of reasoning some features might benefit from values implementing Ord somehow so that cargo can deduce minimal requirements (e.g. off, minimal, full)

  • Mutual exclusion isn't enough in some cases as globals might have to interact with each other in unique ways.

Basically, the only solution that I think would handle all the use cases properly is having a custom trait in the build script that allows defining features and how they adapt to env vars, their environment, as well as each other; and allows them to return an error message if something's wrong (e.g. missing a so).

27

If I create a heterogeneous_tuple crate which defines a trait for dealing with tuples as a list of types, (due to lack of support for vararg generics) I'd need to generate code for 0..N sized tuples and like to have N be configurable because generic code might require 20 to 60 different types (e.g. strongly typed I/O pins on a circuit simulator). Enum globals don't handle that case well, a numeric global would allow the user to set the value to a required maximum which might be less or much more than anticipated preset values.

This is discussed in the Future Possibilities

Mutual exclusion isn't enough in some cases as globals might have to interact with each other in unique ways.

I imagine this would be best left to build.rs or cfgs in code, rather than baking into the design.

28

Perhaps even moving to a zig style build.rs, where IIRC you're calling build APIs, instead of the other way around, in the long term. No idea how zig deals with library builds though...

29

Last time I looked at zig, the answer was that it doesn't. All the zig code is just one big compilation unit driven by the root package. Their @import is a bit more than just a pragma once textual include, but not significantly more.

If a library has special build considerations, you import their build code.

30

That's... Probably the right choice, but makes introducing new build config deep in a tree pretty rough.

31

Coming from Platform target-specific features · Issue #1197 · rust-lang/cargo · GitHub

wgpu (a general purpose GPU library) supports multiple backends: GLES, DX11/12, Vulkan and Metal. Unfortunately they are not all equal, e.g. Vulkan is available on Linux and Windows, but on MacOS it requires MoltenVK (a compatibility layer).

Currently wgpu enables the Vulkan backend by default on Linux and Windows but not on MacOS. To let users disable those, we need features that can be disabled by using default-features = false but are enabled by default. So the solution here is to create two features, e.g. vulkan-native and vulkan-moltenvk. But both would use the same dependencies, so when using default = ["vulkan-native"] it would pull in all dependencies on MacOS as well, even though the desire here is to not enable Vulkan by default on MacOS.

Unfortunately the only workaround I'm aware of, is to create a crate, e.g. wgpu-default-features, that enables the desired crate features by default on each platform by using target.'cfg(...)'.dependencies and then using default = ["dep:wgpu-default-features"]. Not only is it pretty annoying to have to maintain a workaround crate, but in wgpu cfg guards now have to test for vulkan-native, vulkan-moltenvk and default.

I also described in a follow up post (Platform target-specific features · Issue #1197 · rust-lang/cargo · GitHub) how this proposal in its current form isn't able to address this issue:

Specifically the proposal mentions but leaves out:

  • Target-specific set-globals, which would have to play well with default as well.
  • Multi-valued globals, which Wgpu could probably live without, but would still introduce some complexity, having to play around with multiple globals that do the same thing depending on the target or if using the default.

Though it is true that current Wgpu crate features would be best served if controlled by the final artifact only, this isn't always the case. E.g. game engines like Bevy may want to expose different defaults then Wgpu or be able to enable certain Wgpu features when enabling some other Bevy features.

So the problem in this case is that "globals" is not complementary to crate features, but a completely separate system.

32

I'm worried about mutually exclusive global features leading to dependency conflicts. I think that an alternative that should be mentioned in the list should be generics, especially for the multiple-backends use-case. sqlx and gfx-hal are two great examples of supporting multiple backends through generics.

The main benefit of generics is that they offer local mutual exclusivity, handled by Rust's scoping rules. This is great as it means that you can mix incompatible features in the same program (but in different scopes) and it's a huge strength. For example, I can usually use only Postgres with sqlx, but it is possible to have both Postgres and Sqlite support inside the same program, and use them in different parts. With the async runtime example, I hope that a similar solution with a generic abstraction (especially with recent progress with async in trait) so I can use libs depending on specific runtime or runtime-agnostic ones without getting dependency conflicts.

I see how useful it can be for truly global config such as linked libs, but using it for multiple backends seems like a step backwards.

1 Like
33

Could you expand on the dependency conflict concern?

Global features can only be specified by the defining library and then be overridden by the current build process itself / final artifacts.

34

To be more clear, what worries me are intermediate dependencies expecting particular globals. If we have a lib foo with two backends (foo_a, foo_b) that can be picked by the main executable my_exe, how do we deal with libs bar_a depending on foo with backend foo_a and quux_b depending on foo with backend foo_b.

It would correspond to the following package definitions:

# Some lib with support for multiple backends
[package]
name = "foo"

[globals.foo_backend]
description = "Backend to use for foo's impl"
values = ["foo_a", "foo_b"]
default = "foo_a"

# intermediate lib depending on `foo`, and expecting it to have the backend `foo_a`
[package]
name = "bar_a"

[dependencies]
foo = "..."
# No way to specify in the manifest that it expects `globals.foo_backend = "foo_a"`, but it fails at compile time if it's not the right value

# intermediate lib depending on `foo`, and expecting it to have the backend `foo_b`
[package]
name = "quux_b"

[dependencies]
foo = "..."
# No way to specify in the manifest that it expects `globals.foo_backend = "foo_b"`, but it fails at compile time if it's not the right value

# Main executable
[package]
name = "my_exe"
set-globals = {foo_backend = ["foo_a"]} # or should it be `foo_b`?

[dependencies]
bar_a = "..."
quux_b = "..."
# And now we get in a situation where `bar_a` and `quux_b` are in conflict because they expect different backends for `foo_backend`

To summarize, what worries me are dependencies sharing a base lib but having different expectations about its global features.

To give a less abstract example, sqlx allows you to enable different DBs without conflict, so you can connect to both Sqlite and Postgres in the same exe and transfer data, that's nice. However, they also have mutually exclusive features for their TLS impl and async runtime. At work we have a lib db_helpers reusing sqlx but expecting it to use tokio. If there's some other lib with different functions that may be useful but expects async-std for sqlx, I have to pick one or the other helper lib but can't use both.

In a way, the problem of having exclusive features and incompatible libs because of this already exists and this RFC just helps clarifying it. However, the current guidance is to ensure features are additive, and I was hoping that progress in the language would help remove the cases where exclusive features are used currently.

To give a bit more context, I really like global feature unification in general, but there are some situations where it's somewhat limiting. A recent example I have in mind is serde where it uses cargo features to pick between having a stable order or not for maps, or to support arbitrary float precision or not. Because of this, it raises a situation where all your code can use the unordered impl, but then if pull a dep enabling stable ordering for a small feature; then your whole codebase starts using stable ordering. It's still working, but there's less control over what feature is used where.

I try personally to use features only to enable new code and never to swap impls (I avoid negations in cfg). Instead, implementations choices are "tunneled up" through generics and can be scoped as needed. This style is a bit more verbose, but it allows to go pretty far while preserving locality of impl choices. Because of this experience, I see generics as an alternative to cargo features for the use case of backend selection.

35

My expectation for mutually exclusive features would be (assuming a crate C with mutually exclusive features):

  • If nothing in the dependency graph picks a feature, use the default value specified by C.
  • Anything "higher" in the dependency graph overrides something "beneath" it in the dependency graph (something it depends on, directly or indirectly).
  • If one crate depending (directly or indirectly) on C picks a feature, use that one.
  • If two or more crates depending (directly or indirectly) on C pick the same features, use that one.
  • If two or more crates depending (directly or indirectly) on C pick different features, then give a compilation error unless something above all of those crates chooses a single feature.
  • Ultimately, the thing "above" everything is the top-level binary crate, so if nothing else reconciles the features, the top-level binary crate has to pick a value for the feature.

Or, flipping that on its head and turning it into more of an algorithm:

  • At the end of the algorithm, there must be only one value for the global feature, for every crate in the dependency graph.
  • Every crate depending (directly or indirectly) on the crate with the global feature has a "proposed value" for the global feature. (This is not necessarily the final value.) The proposed value may be None.
  • The crate defining the feature can set a default, in which case that's that crate's proposed value for the global feature. If the crate defining the feature doesn't set a default, then its proposed value is None.
  • Any crate that explicitly defines a value for the global feature has that as its proposed value.
  • Any crate that doesn't explicitly define a value for the global feature checks the proposed values from all its dependencies. If those are all identical, the crate inherits that same proposed value. If any differ, the crate gets a proposed value of None.
  • If the top-level binary crate has a proposed value that is not None, that becomes the final value.
  • If the top-level binary crate has a proposed value of None, emit an error, print the crates in the crate graph that have proposed values, and say that the top-level crate has to define the value itself.

In other words, if your dependencies agree or don't care, you don't have to specify and can inherit a default; if your dependencies disagree then you have to specify yourself.

36

I assumed global features could only be set in binary crates.

37

That would prevent libraries from encapsulating their usage of a library and not unnecessarily making users of that library deal with decisions about its internals.

If I'm first starting with a new framework, I'm not going to want to make a decision about which TLS library it uses or what zlib backend it uses before I can even write "hello world". I want to rely on the framework, or the libraries used by the framework, to pick reasonable defaults.

Later on, I may have specific needs and very much want to customize that, and I'll be very happy that I can.

2 Likes
38

I think it needs to be carefully considered how globals interact with cross-compilation and workspaces. If they're really global, they could be fragile similar to how resolver = "1" was breaking no-std crates.

Different platforms have different linking requirements, e.g. MUSL and WASM have different needs than macOS or Ubuntu, so globals need to support per-target per-sys-crate settings.

The difference is especially tricky in cross-compilation where build.rs and proc macros may require different settings. If globals are really global, then if I set e.g. async-backend = "io_uring" then such project won't be able to cross-compile from Windows or macOS if a build script ends up depending on this backend. OTOH a setting like 3d-renderer = "apple-metal" could need to preprocess shaders/textures at build time, and the host's build script would need to use target's setting instead.

Unification of globals in a workspace might also be a problem. Imagine an SQL schema migration tool that supports multiple databases. It could have schema-migrator-pg and schema-migrator-mysql bin crates in a workspace. These crates could depend on some db helper tool that forces them to choose an exclusive db backend option, which complicates their co-existence in the workspace.

4 Likes
39

You're right, for at least some uses people may want to be able to set a global feature's value inside a target.this-target-triple section or a target.cfg(...) section.

40

This would avoid the verbosity of specifying that every single dependency should build with serde support, but, it would also make build times worse for the crates that don't actually need serde support (even though I'm using serde elsewhere).

But this could be something included in default features, so there's a way to opt out, with default-features = false.

2 Likes