Fleshing out libstd "scenarios"


#14

In the subtractive case, the compiler won’t let you build until you’ve subtracted all the parts that the emscripten port doesn’t support.

The problem is that this doesn’t axiomatize: What if I’m using a crate that provides RDMA, and I want to build “without infiniband” (for any number of reasons - licensing, not having the hardware, etc)?

The only way what you propose could work is if the set of scenarios is closed, and only extensible by rustc, in the definition of target specs. I feel that would cripple scenarios, which could (IMO) fill a desperately needed role of allowing root crates to limit Cargo’s additive feature behavior. See also, a couple of issues recently where the lack of such an ability has made Cargo considerably less ergonomic for crate authors.


#15

The ability for the root crate to enable and disable scenarios/features and have that control how upstream crates are compiled would be amazing. Imagine being able to disable the openssl scenario without having to manually go through every single dependency to ensure it doesn’t forget to disable the default features of some other dependency so it can disable the default features of yet another dependency so that it doesn’t depend on the openssl crate. Features are great for enabling required functionality, but are terrible for optional functionality.


#16

I’m afraid I don’t know what the relationship between RDMA and infiniband is so I have a hard time visualizing what you are describing. I might imagine the RDMA crate either does or does not declare the infiniband scenario, and the downstream crate does or does not consume it.

The way I’ve imagined scenarios is as a relatively simple effect system propagated through the dag. For any compilation configuration, a crate may or may not introduce a scenario, which is propagated downstream. Downstream crates must then match, for the compilation configuration, the set of scenarios they require with the scenarios present in the world. (Hm, to that end I might expect scenarios to be declared in Cargo.toml, not in source, and enforced entirely within cargo).


#17

To be honest I didn’t elaborate because I don’t really remember exactly what was happening (it’s been a long time, and since the library was in heavy development I didn’t document all the changes correctly).

I think the biggest problem is that users were trying to use functions that are present in the documentation without realizing that they needed to tweak their Cargo.toml to enable the corresponding feature.

In the case of glium, the problem is that the error that the user would get is that the function simply doesn’t exist. If you integrate “scenarios” in the core of Rust, that would be solved by instead returning an error saying that they need to enable the corresponding scenario.


#18

(Disclaimer: I don’t write libraries and don’t use cargo in non-trivial ways.) Cargo features don’t have values? Something like “USE_OPENSSL=0”? Because what you are describing sounds very similar to this thing from cmake

target_compile_definitions(my_root_crate PUBLIC USE_OPENSSL=0)

#19

RDMA is a general term for “remote direct memory access”. There are a number of technologies this can be done over, including Fibre Channel, Infiniband, and Ethernet.

I’m not clear on what you mean by “consume” here - it sounds like you intend these to propagate in the same direction and manner as features (dependencies declare them, direct dependents enable/disable them), and are arguing in favor of a “default on, opt out” semantics.

In my opinion, this would not work well: That propagation manner is insufficient (dependencies declaring them mandates direct dependents enable/disable them due to namespacing issues; if direct dependents control them, they’re isomorphic to features; if they’re isomorphic to features I see minimal benefit), and opt out is dangerous with any other propagation manner (because if I’m building a kernel, I know what is available, but I don’t know what crazy things my Nth-level dependencies might want to be available).

They do not - they are all boolean, and are enabled by cargo if any dependent requires them. This is why negative-polarity options (no_foo) are dangerous/invalid.


#20

No, cargo features don’t have values. They are either enabled or disabled.

The problem is that some dependency might depend on hyper without disabling the default features, effectively forcing hyper to always enable the openssl feature, despite hyper and that dependency both being able to work without that feature. Once enabled by a downstream dependency, that feature is stuck enabled and the user has no control over it unless that dependency provides its own features to enable/disable the features of its dependencies (and all the way transitively until you reach the root crate) which is hell. If even one dependency fails to provide this, everything is ruined unless he user or root crate vendors their own version of that dependency to disable the default features of hyper and have its own feature to enable the openssl feature of hyper. For complex iron projects this process might have to be repeated for multiple dependencies, and can get quite hairy when the problematic dependency is several layers deep.

What Cargo needs is several things:

  1. A way to distinguish between features that are required and optional features that are nice to have.
  2. A way for upstream crates to tell downstream crates which features are actually enabled, so they can take advantage of optional features when they are enabled.
  3. A way for the user or root crate to enable and disable those optional features.

As a bonus, it would be really cool if the build script for a crate could choose which optional features to enable.


#21

What you want is a “program-composable” way of showing that code does not do a thing - if every individual part of your code is malloc-free, then your entire program is malloc-free (as opposed to ways that show an individual function of a malloc-containing program is malloc-free, aka “purity”).


#22

Have we just been using the different terminology for the same thing the whole time :)? To me this is a) better namespacing features with the crate they are defined in, and b) allowing downstream crates (especially root, but perhaps not limited) to “veto” features.


#23

I think this may actually be an important point to work out. My interpretation of this is that you’d like crates to declare which platforms they support, right? If not, then you can ignore the rest of this, but if so, I’m curious! So on one hand this is nice where if I explicitly support a platform I can’t accidentally regress support. On the other hand if I have to explicitly support a platform then adding a new platform means the whole world has to be updated to work with it. (just an interesting tradeoff I think should be considered)

I’m also curious about the implications for subtractive scenarios. Let’s say that your crate works on both emscripten and on Windows. When you’re on emscripten you avoid threads but on Windows you use them as a performance optimization. What would the scenario opt-in look like here? Are you thinking you’d have something like:

#![cfg_attr(target_os = "emscripten", scenario(without(threads)))]

I’d personally prefer to avoid mixing #[cfg] and #[scenario] like this but I’m curious what you think!


Ok, thanks for the info! Sounds like high quality documentation and error messages are critical to this features success, in any rate.


@retep998

To be clear, I don’t think scenarios is suitable for the winapi use case you’ve outlined. The intention here (at least for the standard library) is to allow access to APIs that are otherwise available by default. Now this might mean that the scenario feature above isn’t as ambitious as it could be, but as I’m thinking of it so far at least it does not cover the use case of selecting what the API of a crate should be (e.g. disjoint sets).


#24

Just chiming in with some stream of consciousness:

  1. I wish I could tag feature-flagged modules/items/whatever with a message, or even the compiler has a default message like “did you enable ?” e.g. having a hint here would be nice, but maybe it’s impractical:
error[E0432]: unresolved import `elf::strtab::Strtab`
--> src/image.rs:19:5
 |
19 | use elf::strtab::Strtab;
 |     ^^^^^^^^^^^^^^^^^^^ Could not find `strtab` in `goblin::elf64`

  1. I like the idea of a global namespace for certain feature flags, like “std” I’m noticing some people are spontaneously adopting for allowing no_std situations, but I don’t like the idea of global namespaces in general for many reasons listed above, but also the whole resource squatting thing if that hasn’t been mentioned. Must be very careful with this.
  2. I love/love/love the idea of documentation illustrating what/when/why/how it is disabled. This seems like an oversight that it isn’t already incorporated
  3. The more I think about negative polarities in feature flags with current behavior of cargo, the more it scares me. I’m quite grateful @eternaleye pointed out the error of my ways, but I’m not sure how to prevent authors from doing it…
  4. So I’m working on a new archive api, and it uses the std and endian_fd feature flags. I’d like users to know this automatically (i.e., in documentation) without my having to type it. I almost wish feature-flags themselves were simply documented/documentable in the top crates documentation…

Maybe some or all of these issues will be resolved with scenarios, not sure. If it improves the documentation landscape on feature flags at all then I’ll be happy :slight_smile:


#25

I’m not sure if I completely understand what #[scenario] does, so I’ll just describe two cases I would like it to solve :slight_smile:

  1. I develop on Mac and test on Linux, but I don’t have Windows machine to test on. I would like Rust to warn me when I accidentally write code that does not compile on Windows.

     #![scenario(unix, windows)]  
        
      // #[cfg(unix)] - oops forgot the attribute
     fn cross_platform_method(f: &File) {
         let fd = f.as_raw_fd(); // I expect: Error: would not compile on `windows`
     }
    
  2. I develop and test on a 64-bit machine, but I want a strong guarantee that my code will never overflow on a 32-bit machine. Currently this very hard to do (mainly because use of usize is unavoidable, the as operator is an unchecked cast, and Into doesn’t work in this context).

    I imagine the scenarios mechanism could be used to state “This code must compile on both 32-bit and 64-bit” and “Only 32-bit and 64-bit architectures are supported” (i.e. 16-bt/128-bit may fail to compile and that’s OK). With this the stdlib could add/remove Into implementations as appropriate, so that as-free code is forced to use TryInto where it could lose precision on 32-bit, and also would opt-in to Into implementations that are currently missing due to hypothetical 16-bit and 128-bit support.


#26

So, I’ve been thinking a bit more about this proposal, and I think there is a way to extend it to give stronger, statically checkable, inter-crate guarantees that address some of the concerns I’ve seen about the proposal as-is.

I’m not sure how feasible, or even internally consistent these idea are, but here is what I got:

  • A crate can define any number of scenarios by name.
  • Scenarios can be transparent aliases for combinations of other scenarios.
  • Said combinations would be logical formulas with the operators any() and all() from #[cfg], but additionally also support either() to express exclusivity.
  • Scenarios defined in different crates that share the same name are distinct from each other.
  • Scenarios can be imported from an extern crate to use them in more than one crate.
  • Every single item in a crate gets “tagged” as belonging to one or more scenarios.
  • To make this more manageable, every crate can declare an “implicit scenario” that gets applied to all items automatically.
  • To make this backwards compatible, the implicit scenario defaults to one defined in std, which represents the currently in the community assumed “desktop” scenario of being cross platform compatible for the major platforms linux, windows and MacOS.
  • Scenario-tagged items transitively check that their definition does not access items for scenarios more constrained than them self.
  • A crate has to tag itself with all scenario combinations it supports, with a warning lint that enforces it in a backwards compatible way by checking all items in the crate. This also the serves as a mechanism for enabling scenarios, by supporting more than the default one.
  • The cargo repo would start rejecting new versions of crates that emit that warning, similar to the “*” version dependency check. This ensures gradual transition to the new system.
  • #[cfg] also gains the properties of a scenario tag, and would be checked in the same way. Enabling them per compiler flag would probably have to implicitly define their scenario, with a warning lint demanding explicitness.
  • The compiler would start remembering #[cfg]d items in some fashion, so that the scenario checker can reason about their existence for the crate-wide check.
  • Different #[cfg] flags on items with the same name would cause a “virtual” scenario tag expressing the constraint “either of these needs to be defined”.

This seems like a somewhat complex system, but by picking the right defaults it should be manageable. (And has to be ignore-able anyway, for backwards compatibility)

Examples

Std

// [crate std]

// Import scenarios.
// Attribute behaves like macro_import for selecting either all or a subset.
#[scenario_import]
extern crate core;

// Declare scenarios "atoms", that might correspond to cfg flags
#![declare_scenario(thread)]
#![declare_scenario(unix)]
#![declare_scenario(windows)]
#![declare_scenario(floats)]
// ...

// Declare scenario aliases
#![declare_scenario(desktop, all(either(unix, windows), thread, /* ... */)))]
#![declare_scenario(emscripten, all(floats, core, /* ... */))))]

// Require explicit annotations for all items in std
#![no_implicit_scenario]

// Restrict the std library to a number of scenarios.
// These check the crate assuming only one of them is active at any given time.
#![crate_scenario(desktop)]
#![crate_scenario(all(desktop, unix))]
#![crate_scenario(all(desktop, windows))]
#![crate_scenario(emscripten)]

// ...

impl File {
    #[scenario(desktop)]
    pub fn new() -> Self;

    #[scenario(all(desktop, unix))]
    pub fn as_raw_fd(&self) -> c_int {
        self.fd
    }

    #[scenario(all(desktop, windows))]
    pub fn as_raw_handle(&self) -> *mut HANDLE {
        self.handle
    }
}


Crate with additional features (optional functionality)

// [crate foo]

// Implicit:
// #![scenario_import] extern crate std;
// #![implicit_scenario(desktop)]
// #![crate_scenario(desktop)]
// #[scenario(desktop)] on each item

#![declare_scenario(extra)]

// Implicitly translated as
// #![crate_scenario(and(desktop, extra))]
// because of the implicit desktop scenario
#![crate_scenario(extra)]

// Implicit translated as #[scenario(all(desktop, extra))]
#[cfg(extra)]
fn feature() {
    // ...
}

Crate with reduced features (emscripten compatibility)

// [crate bar]

// Replace the default "desktop" scenario for this crate
// Implies #![crate_scenario(emscripten)] and
// #[scneario(emscripten)] on all items
#![implicit_scenario(emscripten)]

// Dependency is checked to support the emscripten scenario
extern crate dependency;

// ...

Crate that abstracts over different platforms

// [crate cross_plattform]

// Implicit:
// #![scenario_import] extern crate std;
// #![implicit_scenario(desktop)]
// #![crate_scenario(desktop)]
// #[scenario(desktop)] on each item

// Implicitly:
//    #[scenario(all(desktop, windows))]
// == #[scenario(all(all(either(unix, windows), thread, ...), windows))]
// == #[scenario(all(windows, thread, ...))]
#[cfg(windows)]
fn cross_platform_fn() {
    // ...
}

// Implicitly:
//    #[scenario(all(desktop, unix))]
// == #[scenario(all(all(either(unix, windows), thread, ...), windows))]
// == #[scenario(all(unix, thread, ...))]
#[cfg(unix)]
fn cross_platform_fn() {
    // ...
}

// Both above implicitly define
//    #[scenario(either(all(unix, thread, ...), all(windows, thread, ...)))]
// == #[scenario(all(either(windows, unix), thread, ...)]
// == #[scenario(desktop]
// for the item `cross_platform_fn`

// Implicit #[scenario(desktop)], so check ok.
fn cross_platform_fn_2() {
    cross_platform_fn()
}

Crate that causes various errors


// ERROR: Crate implicitly declares itself as belonging to the desktop scenario,
//        but some of its items are only available for the all(desktop, unix) scenario.
// NOTE:  Consider adding #![crate_scenario(unix)] to optionally support the unix scenario.

// ERROR: Access to item b is only enabled for scenario unix,
//        but a implicitly declare itself for the "desktop" scenario
//        which does not imply unix
// NOTE:  Consider adding #[scenario(unix)] to constrain a() to the unix scenario
// NOTE:  Consider adding a implementation of b for the windows scenario
//        to fulfill the desktop scenario
fn a() {
    b();
}

#[cfg(unix)]
fn b() {
}


Winapi families

This is based on the constraints given here.

// [crate winapi]

#![declare_scenario(WINAPI_FAMILY_PC_APP)]
#![declare_scenario(WINAPI_FAMILY_PHONE_APP)]
#![declare_scenario(WINAPI_FAMILY_SYSTEM)]
#![declare_scenario(WINAPI_FAMILY_SERVER)]
#![declare_scenario(WINAPI_FAMILY_DESKTOP_APP)]

#![crate_scenario(either(WINAPI_FAMILY_PC_APP,
                         WINAPI_FAMILY_PHONE_APP,
                         WINAPI_FAMILY_SYSTEM,
                         WINAPI_FAMILY_SERVER,
                         WINAPI_FAMILY_DESKTOP_APP))]
// [crate winapi_consuming_lib]

#[scenario_import]
extern crate winapi;

#![crate_scenario(either(WINAPI_FAMILY_PHONE_APP, WINAPI_FAMILY_DESKTOP_APP))]

#[cfg(WINAPI_FAMILY_PHONE_APP)]
fn cross_platform_fn() {}

#[cfg(WINAPI_FAMILY_DESKTOP_APP)]
fn cross_platform_fn() {}

// [crate binary]

#[scenario_import]
extern crate winapi;
extern crate winapi_consuming_lib;

#![crate_scenario(WINAPI_FAMILY_PHONE_APP)]

What makes this example a bit tricky is that this scenario system only checks for validity, but can not trigger configured builds. That is cargos domain, so the top-level selection of the family would probably have to happen with a cargo feature that gets passed through to all upstream crates up to winapi itself. (Not sure if the current feature system makes this feasible though).


#27

This would be terribly impractical and unfeasible. Expecting every single crate in all the chains from the root crate to winapi to pass along a feature for which family to select is a nightmare. There needs to be a way for the user or root crate to select a family without any dependencies along that chain having to care. At most, crates which depend upon winapi directly should be informed of the choice of family.


#28

A potentially relevant use case which occurred to me: to what extent could scenarios be used to solve the problems of single-source GPU-compute support? One basic and obvious difference this would have relative to what has been envisioned so far is that not only library APIs, but also language features would have to be similarly gated. (But maybe this is at least vaguely similar to things like floating point support?)

I’m not personally very familiar with this area so I’m not the right person to ask for further details about this, but I do have a few physicist friends who would be really interested in this kind of thing being possible.


#29

@retep998

Can winapi configure itself automatically on compilation, or are there more than one possible supported family per platform so that there needs to be some selection somewhere?


#30

@Kimundi Right now Rust hasn’t even come to a decision on whether each family warrants a different target yet. Suffice to say, on a given PC running Windows, you can run an application compiled for any of those families (although phone apps need to be run inside the windows phone emulator, but they’re still technically x86 due to the emulator using hyper-v). For a given application, all libraries and crates need to be compiled for that family, including std and friends, so it is compatible with one target per family approach. However, because nobody from the Rust team has given me any sort of answer on how we’ll be proceeding with this, I can’t give any sort of definitive answer.


Refactoring std for ultimate portability
#31

One thing that came up in the lang meeting: we can teach the compiler about items that have been cfg-ed out, including other definitions of the same item.

While cfg-polymorphic compilation would be difficult because the choices would have to end up in types and in the MIR, reasoning about the conditional existence of public definitions and tainting other public definitions which use them, transitively, would be a fairly routine analysis.

A tool like rustdoc could display all possible public definitions, and provide links to docs of different platform-specific configurations of the same crate. The cfg dependencies observed can be used to check that these docs aren’t missing.

For definitions that might not survive codegen / linking, but otherwise type-check fine, we could just not cfg them out, but keep some annotation around to prevent ever instantiating them accidentally (mostly thinking of imperfect compiler checks here).


Getting explicit SIMD on stable Rust
#32

It might be useful to have an io scenario, which, if turned off, would disable the blocking io primitives in std::io and std::net. Applications built on an async IO framework like tokio could disable io, making it impossible for them to perform blocking IO without doing something like C FFI or direct syscalls. This can reduce the problem async IO has of people accidentally blocking the event loop.


#33

I’ve now posted an RFC based loosely on these ideas, but significantly evolved.