Fleshing out libstd "scenarios"

What you want is a "program-composable" way of showing that code does not do a thing - if every individual part of your code is malloc-free, then your entire program is malloc-free (as opposed to ways that show an individual function of a malloc-containing program is malloc-free, aka "purity").

Have we just been using the different terminology for the same thing the whole time :)? To me this is a) better namespacing features with the crate they are defined in, and b) allowing downstream crates (especially root, but perhaps not limited) to "veto" features.

I think this may actually be an important point to work out. My interpretation of this is that you'd like crates to declare which platforms they support, right? If not, then you can ignore the rest of this, but if so, I'm curious! So on one hand this is nice where if I explicitly support a platform I can't accidentally regress support. On the other hand if I have to explicitly support a platform then adding a new platform means the whole world has to be updated to work with it. (just an interesting tradeoff I think should be considered)

I'm also curious about the implications for subtractive scenarios. Let's say that your crate works on both emscripten and on Windows. When you're on emscripten you avoid threads but on Windows you use them as a performance optimization. What would the scenario opt-in look like here? Are you thinking you'd have something like:

#![cfg_attr(target_os = "emscripten", scenario(without(threads)))]

I'd personally prefer to avoid mixing #[cfg] and #[scenario] like this but I'm curious what you think!


Ok, thanks for the info! Sounds like high quality documentation and error messages are critical to this features success, in any rate.


@retep998

To be clear, I don't think scenarios is suitable for the winapi use case you've outlined. The intention here (at least for the standard library) is to allow access to APIs that are otherwise available by default. Now this might mean that the scenario feature above isn't as ambitious as it could be, but as I'm thinking of it so far at least it does not cover the use case of selecting what the API of a crate should be (e.g. disjoint sets).

Just chiming in with some stream of consciousness:

  1. I wish I could tag feature-flagged modules/items/whatever with a message, or even the compiler has a default message like ā€œdid you enable ?ā€ e.g. having a hint here would be nice, but maybe itā€™s impractical:
error[E0432]: unresolved import `elf::strtab::Strtab`
--> src/image.rs:19:5
 |
19 | use elf::strtab::Strtab;
 |     ^^^^^^^^^^^^^^^^^^^ Could not find `strtab` in `goblin::elf64`

  1. I like the idea of a global namespace for certain feature flags, like ā€œstdā€ Iā€™m noticing some people are spontaneously adopting for allowing no_std situations, but I donā€™t like the idea of global namespaces in general for many reasons listed above, but also the whole resource squatting thing if that hasnā€™t been mentioned. Must be very careful with this.
  2. I love/love/love the idea of documentation illustrating what/when/why/how it is disabled. This seems like an oversight that it isnā€™t already incorporated
  3. The more I think about negative polarities in feature flags with current behavior of cargo, the more it scares me. Iā€™m quite grateful @eternaleye pointed out the error of my ways, but Iā€™m not sure how to prevent authors from doing itā€¦
  4. So Iā€™m working on a new archive api, and it uses the std and endian_fd feature flags. Iā€™d like users to know this automatically (i.e., in documentation) without my having to type it. I almost wish feature-flags themselves were simply documented/documentable in the top crates documentationā€¦

Maybe some or all of these issues will be resolved with scenarios, not sure. If it improves the documentation landscape on feature flags at all then Iā€™ll be happy :slight_smile:

Iā€™m not sure if I completely understand what #[scenario] does, so Iā€™ll just describe two cases I would like it to solve :slight_smile:

  1. I develop on Mac and test on Linux, but I donā€™t have Windows machine to test on. I would like Rust to warn me when I accidentally write code that does not compile on Windows.

     #![scenario(unix, windows)]  
        
      // #[cfg(unix)] - oops forgot the attribute
     fn cross_platform_method(f: &File) {
         let fd = f.as_raw_fd(); // I expect: Error: would not compile on `windows`
     }
    
  2. I develop and test on a 64-bit machine, but I want a strong guarantee that my code will never overflow on a 32-bit machine. Currently this very hard to do (mainly because use of usize is unavoidable, the as operator is an unchecked cast, and Into doesnā€™t work in this context).

    I imagine the scenarios mechanism could be used to state ā€œThis code must compile on both 32-bit and 64-bitā€ and ā€œOnly 32-bit and 64-bit architectures are supportedā€ (i.e. 16-bt/128-bit may fail to compile and thatā€™s OK). With this the stdlib could add/remove Into implementations as appropriate, so that as-free code is forced to use TryInto where it could lose precision on 32-bit, and also would opt-in to Into implementations that are currently missing due to hypothetical 16-bit and 128-bit support.

1 Like

So, Iā€™ve been thinking a bit more about this proposal, and I think there is a way to extend it to give stronger, statically checkable, inter-crate guarantees that address some of the concerns Iā€™ve seen about the proposal as-is.

Iā€™m not sure how feasible, or even internally consistent these idea are, but here is what I got:

  • A crate can define any number of scenarios by name.
  • Scenarios can be transparent aliases for combinations of other scenarios.
  • Said combinations would be logical formulas with the operators any() and all() from #[cfg], but additionally also support either() to express exclusivity.
  • Scenarios defined in different crates that share the same name are distinct from each other.
  • Scenarios can be imported from an extern crate to use them in more than one crate.
  • Every single item in a crate gets ā€œtaggedā€ as belonging to one or more scenarios.
  • To make this more manageable, every crate can declare an ā€œimplicit scenarioā€ that gets applied to all items automatically.
  • To make this backwards compatible, the implicit scenario defaults to one defined in std, which represents the currently in the community assumed ā€œdesktopā€ scenario of being cross platform compatible for the major platforms linux, windows and MacOS.
  • Scenario-tagged items transitively check that their definition does not access items for scenarios more constrained than them self.
  • A crate has to tag itself with all scenario combinations it supports, with a warning lint that enforces it in a backwards compatible way by checking all items in the crate. This also the serves as a mechanism for enabling scenarios, by supporting more than the default one.
  • The cargo repo would start rejecting new versions of crates that emit that warning, similar to the ā€œ*ā€ version dependency check. This ensures gradual transition to the new system.
  • #[cfg] also gains the properties of a scenario tag, and would be checked in the same way. Enabling them per compiler flag would probably have to implicitly define their scenario, with a warning lint demanding explicitness.
  • The compiler would start remembering #[cfg]d items in some fashion, so that the scenario checker can reason about their existence for the crate-wide check.
  • Different #[cfg] flags on items with the same name would cause a ā€œvirtualā€ scenario tag expressing the constraint ā€œeither of these needs to be definedā€.

This seems like a somewhat complex system, but by picking the right defaults it should be manageable. (And has to be ignore-able anyway, for backwards compatibility)

Examples

Std

// [crate std]

// Import scenarios.
// Attribute behaves like macro_import for selecting either all or a subset.
#[scenario_import]
extern crate core;

// Declare scenarios "atoms", that might correspond to cfg flags
#![declare_scenario(thread)]
#![declare_scenario(unix)]
#![declare_scenario(windows)]
#![declare_scenario(floats)]
// ...

// Declare scenario aliases
#![declare_scenario(desktop, all(either(unix, windows), thread, /* ... */)))]
#![declare_scenario(emscripten, all(floats, core, /* ... */))))]

// Require explicit annotations for all items in std
#![no_implicit_scenario]

// Restrict the std library to a number of scenarios.
// These check the crate assuming only one of them is active at any given time.
#![crate_scenario(desktop)]
#![crate_scenario(all(desktop, unix))]
#![crate_scenario(all(desktop, windows))]
#![crate_scenario(emscripten)]

// ...

impl File {
    #[scenario(desktop)]
    pub fn new() -> Self;

    #[scenario(all(desktop, unix))]
    pub fn as_raw_fd(&self) -> c_int {
        self.fd
    }

    #[scenario(all(desktop, windows))]
    pub fn as_raw_handle(&self) -> *mut HANDLE {
        self.handle
    }
}


Crate with additional features (optional functionality)

// [crate foo]

// Implicit:
// #![scenario_import] extern crate std;
// #![implicit_scenario(desktop)]
// #![crate_scenario(desktop)]
// #[scenario(desktop)] on each item

#![declare_scenario(extra)]

// Implicitly translated as
// #![crate_scenario(and(desktop, extra))]
// because of the implicit desktop scenario
#![crate_scenario(extra)]

// Implicit translated as #[scenario(all(desktop, extra))]
#[cfg(extra)]
fn feature() {
    // ...
}

Crate with reduced features (emscripten compatibility)

// [crate bar]

// Replace the default "desktop" scenario for this crate
// Implies #![crate_scenario(emscripten)] and
// #[scneario(emscripten)] on all items
#![implicit_scenario(emscripten)]

// Dependency is checked to support the emscripten scenario
extern crate dependency;

// ...

Crate that abstracts over different platforms

// [crate cross_plattform]

// Implicit:
// #![scenario_import] extern crate std;
// #![implicit_scenario(desktop)]
// #![crate_scenario(desktop)]
// #[scenario(desktop)] on each item

// Implicitly:
//    #[scenario(all(desktop, windows))]
// == #[scenario(all(all(either(unix, windows), thread, ...), windows))]
// == #[scenario(all(windows, thread, ...))]
#[cfg(windows)]
fn cross_platform_fn() {
    // ...
}

// Implicitly:
//    #[scenario(all(desktop, unix))]
// == #[scenario(all(all(either(unix, windows), thread, ...), windows))]
// == #[scenario(all(unix, thread, ...))]
#[cfg(unix)]
fn cross_platform_fn() {
    // ...
}

// Both above implicitly define
//    #[scenario(either(all(unix, thread, ...), all(windows, thread, ...)))]
// == #[scenario(all(either(windows, unix), thread, ...)]
// == #[scenario(desktop]
// for the item `cross_platform_fn`

// Implicit #[scenario(desktop)], so check ok.
fn cross_platform_fn_2() {
    cross_platform_fn()
}

Crate that causes various errors


// ERROR: Crate implicitly declares itself as belonging to the desktop scenario,
//        but some of its items are only available for the all(desktop, unix) scenario.
// NOTE:  Consider adding #![crate_scenario(unix)] to optionally support the unix scenario.

// ERROR: Access to item b is only enabled for scenario unix,
//        but a implicitly declare itself for the "desktop" scenario
//        which does not imply unix
// NOTE:  Consider adding #[scenario(unix)] to constrain a() to the unix scenario
// NOTE:  Consider adding a implementation of b for the windows scenario
//        to fulfill the desktop scenario
fn a() {
    b();
}

#[cfg(unix)]
fn b() {
}


Winapi families

This is based on the constraints given here.

// [crate winapi]

#![declare_scenario(WINAPI_FAMILY_PC_APP)]
#![declare_scenario(WINAPI_FAMILY_PHONE_APP)]
#![declare_scenario(WINAPI_FAMILY_SYSTEM)]
#![declare_scenario(WINAPI_FAMILY_SERVER)]
#![declare_scenario(WINAPI_FAMILY_DESKTOP_APP)]

#![crate_scenario(either(WINAPI_FAMILY_PC_APP,
                         WINAPI_FAMILY_PHONE_APP,
                         WINAPI_FAMILY_SYSTEM,
                         WINAPI_FAMILY_SERVER,
                         WINAPI_FAMILY_DESKTOP_APP))]
// [crate winapi_consuming_lib]

#[scenario_import]
extern crate winapi;

#![crate_scenario(either(WINAPI_FAMILY_PHONE_APP, WINAPI_FAMILY_DESKTOP_APP))]

#[cfg(WINAPI_FAMILY_PHONE_APP)]
fn cross_platform_fn() {}

#[cfg(WINAPI_FAMILY_DESKTOP_APP)]
fn cross_platform_fn() {}

// [crate binary]

#[scenario_import]
extern crate winapi;
extern crate winapi_consuming_lib;

#![crate_scenario(WINAPI_FAMILY_PHONE_APP)]

What makes this example a bit tricky is that this scenario system only checks for validity, but can not trigger configured builds. That is cargos domain, so the top-level selection of the family would probably have to happen with a cargo feature that gets passed through to all upstream crates up to winapi itself. (Not sure if the current feature system makes this feasible though).

1 Like

This would be terribly impractical and unfeasible. Expecting every single crate in all the chains from the root crate to winapi to pass along a feature for which family to select is a nightmare. There needs to be a way for the user or root crate to select a family without any dependencies along that chain having to care. At most, crates which depend upon winapi directly should be informed of the choice of family.

A potentially relevant use case which occurred to me: to what extent could scenarios be used to solve the problems of single-source GPU-compute support? One basic and obvious difference this would have relative to what has been envisioned so far is that not only library APIs, but also language features would have to be similarly gated. (But maybe this is at least vaguely similar to things like floating point support?)

Iā€™m not personally very familiar with this area so Iā€™m not the right person to ask for further details about this, but I do have a few physicist friends who would be really interested in this kind of thing being possible.

1 Like

@retep998

Can winapi configure itself automatically on compilation, or are there more than one possible supported family per platform so that there needs to be some selection somewhere?

@Kimundi Right now Rust hasnā€™t even come to a decision on whether each family warrants a different target yet. Suffice to say, on a given PC running Windows, you can run an application compiled for any of those families (although phone apps need to be run inside the windows phone emulator, but theyā€™re still technically x86 due to the emulator using hyper-v). For a given application, all libraries and crates need to be compiled for that family, including std and friends, so it is compatible with one target per family approach. However, because nobody from the Rust team has given me any sort of answer on how weā€™ll be proceeding with this, I canā€™t give any sort of definitive answer.

3 Likes

One thing that came up in the lang meeting: we can teach the compiler about items that have been cfg-ed out, including other definitions of the same item.

While cfg-polymorphic compilation would be difficult because the choices would have to end up in types and in the MIR, reasoning about the conditional existence of public definitions and tainting other public definitions which use them, transitively, would be a fairly routine analysis.

A tool like rustdoc could display all possible public definitions, and provide links to docs of different platform-specific configurations of the same crate. The cfg dependencies observed can be used to check that these docs arenā€™t missing.

For definitions that might not survive codegen / linking, but otherwise type-check fine, we could just not cfg them out, but keep some annotation around to prevent ever instantiating them accidentally (mostly thinking of imperfect compiler checks here).

3 Likes

It might be useful to have an io scenario, which, if turned off, would disable the blocking io primitives in std::io and std::net. Applications built on an async IO framework like tokio could disable io, making it impossible for them to perform blocking IO without doing something like C FFI or direct syscalls. This can reduce the problem async IO has of people accidentally blocking the event loop.

Iā€™ve now posted an RFC based loosely on these ideas, but significantly evolved.

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.