libstd provides abstractions for various functionality that anyone would expect in any modern programming language for each supported operating system. However, this is, I feel, majorly inadequate and inflexible because every time we want to add a new OS or environment we need to modify
libstd. Therefore, I propose the following change which may require major modifications to
libstd's internals, hence me posting about it here.
Instead of using per-OS implementations of functions directly within
libstd, I propose the idea of a
hook: a highly specialized library or implementation of certain functions that
libstd requires for a particular operation, such as threads, processes, FS access, etc. Here's how it would work: whenever
libstd requires the use of a system/environment-specific interface, such as access to the filesystem, it will call one of its registered hooks to perform the operation. The hook may be within the main application itself or it may be in a dependency. Hooks are registered by using the
std::hooks module and by calling
std::hooks::register(). (I'm not sure on the prototype of this function yet.) If a hook is not registered for a particular operation, the function that is being attempted returns immediately with a new error,
std::hooks::HookError. Hooks could also be unregistered, making that operation inoperative again, (possibly) allowing smooth transitions from one environment to another (though I don't know if this is even possible), or allowing a program to seamlessly change the operation of a function, by, for example, changing a system call to an improved system call that works better. These
libstd hooks would make it possible for
libstd to be completely independent of the underlying runtime environment and would allow anyone to publish new environment hooks for any operating system or execution environment that exists (or may exist) on any of the supported architectures by Rust/LLVM. This would additionally have the advantage of (possibly) eliminating
no_std entirely. Instead, all one needs to do is write their own hooks for
libstd and they can freely use
libstd anywhere. This, I think, would greatly speed up the development of
libstd, since you don't need to register hooks you don't need, and thereby automatically excluding functionality. The compiler, then, could (and would) most likely optimize away code that immediately returns with error conditions due to hooks not being present, or code that is never called.
This idea would also make it easier to add new functionality to
libstd that depends on the external environment. You just add a new hook and off you go. You don't need to care about whether an OS or environment supports it because if someone wants it, they'll hook it and use it.
In order for this to work, we may most likely need to take the existing OS implementations and split them off into a separate set of crates (or maybe a single crate if that is desirable) to eliminate the dependency on an existing OS.
We could stick with what we have right now if this just isn't feasible.
- More overhead: those who want to hook into new environments and get the full standard library experience will need to write a lot of boilerplate code to enable all of the functionality, which will require them to thoroughly understand the execution environment they're running in. This transfers the burden of "understand the environment" from the library team and anyone who contributes new OS support implementations to
libstdto those who actually need the functionality, which some may find undesirable.
- Difficulty may be hard:
libstdis quite large, as it imports
liballoc, and a lot of functionality may need to be removed or altered for this to work properly.
- Modularity and portability:
libstdwill be completely generic across any operating system or runtime environment, regardless of how it works underneath. Contributors or maintainers of the library will no longer need to care about implementation details of how a given operating system handles file permissions or network connections, for example, because that will be abstracted away.
- Simplicity: this change may make it far easier for people to understand
libstd. It may also lower the barrier of entry to those who want to add new things to the library in future.
The UEFI firmware environment is an excellent example of how this hooks mechanism could be utilized. Currently, standard library implementations are found on crates.io and other external registries. The problem with these implementations is that they are effectively duplicates of original work by the library team. This does not follow the DRY principal, and causes major code duplication. It would be much better if
libstd was generic in this manner because then everyone would be using the same standard library and would have all the guarantees that entails, including receiving any updates whenever
rustc is updated. Currently, maintainers of these custom
libstd implementations must either copy or re-implement the functionality that
libstd provides, depending on the license of the crate in question.
As an example, a UEFI environment hook crate for this hypothetical
libstd would need to just register the FS hook using either
EFI_FILE_PROTOCOL, depending on what functionality they want. This could similarly be applied to other standard library functions:
- Environment variables: NVRAM variables via
- Threads: Emulation via
What do you guys think of this proposal? I'm not sure about the definitions yet, but this is just something I wanted to bounce off you guys and see if its even possible to begin with since it was bouncing around my head and wouldn't go away.