Secretly and behind the scenes more and more Rust code lives a life before main. What is life before main? It's code that executes before the rust runtime has been initialized in main. This is something that Rust does not support but because the world needs to support C++ static initializers and __attribute__((constructor)) in C, it's also possible to get Rust code to execute this way.
The popular crate for this is ctor. Using it is probably not a great idea, but unfortunately there are lots of situations where it's hard to avoid using it. However, even if you are not using it directly, you might be depending on it indirectly. For instance the very popular inventory crate uses it behind the scenes.
There are a handful of really important use cases that require something like life before main as there is no hook that crates can use to automatically run something. A good example for this are all types of plugin hooks (think test or benchmark registration) or when you need to initialize C libraries once (eg: the rust curl library does something like this).
There are some alternative ideas currently floating around for how this can work (eg: see dtolnay's linkme) crate which does not involve code execution.
However there is another case to consider. It's today already necessary to run code (and not just provide data) when main runs. The reason for this is that very often you do not control main. A good example is rust-test right now. The main of rusttest is not available which means if you absolutely need to run test before tests run there is no guaranteed way to do this currently.
I was wondering what people think about instead of "life before main" Rust gains a combination of what linkme does to facilitate "common life in main".
So rust code can register a function to be added to a list of functions that will be executed in main (or rather after the runtime is set up but maybe before main is invoked).
Eg:
#[startup]
fn run_me_on_startup() {
// do some stuff here
}
That would give the rust runtime some control over when these functions execute and different targets could control this better. In particular things like rust-test and whatnot might be able to better control how these initializers should run.
I don't think that dlopen for instance should run constructors. That's a major source of main and it also does not seem to be something that needs to be supported. For all the cases where someone loads stuff dynamically there is typically already some API in place anyways that should be called with some state being passed in.
Having read through previous proposals on life before main, but not being particularly versed in the topic, I can at least say that the exact notation used to indicate things that should run before main is probably one of the smaller items needing a solution. That's fine if your goal is simply to reopen discussion, but there's a lot of previous discussion that should probably at least be collected and summarized first.
I would tend to agree that historically life before main was not much of an issue. However recent breakage with the LLVM update and recognition how much of the Rust ecosystem actually depends on this by now I believe has changed this quite significantly.
The last significant discussions about this died a few years ago after which people mostly just ignored it as they used crates like ctor. However the LLVM 13 update broke ctor in a lot of situations and now we have a bigger issue as people started depending (indirectly) on it working.
Forgive my ignorance, but wouldn't "standard lazy types" or the lazy_static crate (which can be used on stable Rust) solve most, if not all, no-life-before-main problems, assuming people actually use them? Are there any situations that using one of these would not address?
No, because that requires active use (eg: you need to access to initialize). The issues that people solve with ctor and co right now are plugin type situations where the pure declaration of something needs to register it implicitly.
This is for instance currently used by crates like inventory. It also comes up with test situations regularly. For instance the test support in procspawn depends on it as there is no other exposed hook to make sure that stuff runs once when tests start up.
The case I'm currently running into is registering a bunch of custom #[bench] functions with a benchmark registry similar to how #[test] currently registers all tests with rust-test.
All these situations are not solvable with lazy_static or similar.
After re-reading your posts and looking at some of the links you posted, if I understand correctly, the problem is that you don't control the main function (like in the case for Rust's testing framework), and you need something to run before the code in main runs, so that it will be available when main runs, but there is no hook you can use to perform this initialization. Is that correct? Mainly just asking out of curiosity to understand the issue, as I don't know that I have anything more that I could contribute to the conversation right now.
To me, that implies that we shouldn't have life before main (in the sense of running code before main), but instead we should have some way to build a 'static slice automatically out of a bunch of instances that were marked automatically.
That helps avoid all the problems that come from running code -- like cycles and ordering -- while still allowing the registration uses.
An unordered attribute like #[startup] will likely run into the same thing. For example you may want to register your plugin automagically, but your plugin may itself need to wait for some of its dependencies to initialize first before it can register itself (this can realistically happen, e.g. a video-conferencing plugin may want to wait for audio and video codecs plugins to register themselves first, and codec plugins may want to wait for hardware acceleration plugins to register first).
Another ordering issue to consider is setup sandboxes and child forks. I have servers that call fork() as the first thing in main, and then set up two different sets of seccomp filters for roles of the child and parent process (including specifically two different ways of reporting errors to sentry, since one of the processes is intentionally forced offline). Life before main would be terrible for this — I'd be running unsandboxed code, and half of the things would be in a wrong process.
The core issue with the traditional ctor approach is just that stuff runs and nobody has any control over it. With just registering functions there is a certain amount of control that can be exposed.
A more generic solution than #[startup] would be to have a #[register(MY_COLLECTION)] similar to how linkme operates. At that point the user gets all collected functions or types of a specific collection and can do with them what the user wants. However I think at all times the equivalent to #[startup] is necessary because of the lack of control over main at the moment.
If we have #[distributed_slice], then I think it's fine to push any framework that takes main away from you to offer a distributed slice of fn() hooks that it runs early in the life of the program. This means that you don't automatically have #[startup] with whatever framework, but it gives the framework control over when and how the startup hooks are ran. (So e.g. they can run after logging is initialized, or w/e.)
Ordering is still an interesting question to answer, but it's now answerable in userspace (e.g. the bevy scheduler) rather than needing to be solved in the compiler. (And there definitely are competing answers as to how to solve startup scheduling.)
(Side note: I really want to see a world where bevy can use distributed slices to construct the world. They've considered and rejected using linkme because of the platform limitations, but if it were built into the compiler directly, it definitely would get reconsidered.)
setenv/getenv are so difficult to make thread-safe because the problem spans Rust and C
are constructors not a case like that?
sometimes you have some control over main, sometimes not
sometimes it is more important that your plugin runs before main
sometimes it is more important that main runs before anything
sometimes main is in Rust, sometimes in C
...the last "sometimes" seems most problematic
you end up trying to fix the world, not just Rust
P.S. #[distributed_slice] on all platforms would have been wonderful..
Having built plugin systems that need to work in shared and static builds (while supporting "builtin" plugins as well), I think some way to just "label" a piece of static data (or function) that gets put into the binary and then have an API to ask the current executable (and its transitively loaded shared libraries) as well as "this specific loaded library" things like "get me all symbols labeled with foo" would work. Now, this would be an unsafe API since going from symbol name to some concrete type is ripe for…abuse, but going from "ha, good luck, hope the linker doesn't screw you over" to "use unsafe to get what you need" is a vast improvement I think. This would allow for more…reasonable initialization routines to be built up from such a primitive.
FWIW, plugins shouldn't require "life before main" and should be able to instead wait for explicit instantiation (both because of the general unorderedness but also because plugins may depend on each other). But there may be other use cases as well. Can we gather a list?
FWIW, in Rust, I've used inventory, but if there were the "get me symbols matching X" primitive (probably provided by a crate given its likely platform availability and looking at linkme, maybe this is it), could similar behavior be implemented? It would be nice to have libloading (or similar) be able to have ways of querying for freshly-loaded information as well.
For example you may want to register your plugin automagically, but your plugin may itself need to wait for some of its dependencies to initialize first before it can register itself (this can realistically happen, e.g. a video-conferencing plugin may want to wait for audio and video codecs plugins to register themselves first, and codec plugins may want to wait for hardware acceleration plugins to register first).
I'm pretty apprehensive of rust adding anything which could make life-before-main more common. However this comment made me think: Couldn't the compiler be smart enough to handle this? You could have a restriction that a crate is only allowed to have a single function annotated #[startup], and the compiler could ensure that the #[startup] functions of dependencies run first.
I think that "gather a list" is the proper solution to the plugin problem, as proposed above.
We could either embed the dependency graph in the collected items, or require the Rust compiler to order the items in "bottom up dependency order" (for anything statically available).
Implementation weeds for plugins
After you have dependency tree ordering, it's easy to borrow the five phase system from Minecraft modding (Initialization: more unsafe than usual, similar to ctor: your dependents have not loaded yet; Early, Normal, Late, Finalize: make no new changes, instead, compact your records of changes already made.)
Please forgive my ignorance, but I've never heard of this mechanism. How would it work? Does the compiler do a pass to determine the space needed for all items in the distributed slice, then statically allocate space for it? Or is it more like a vector, requiring an allocator to work?
My understanding is that it makes a linker section for the relevant data and uses a linker script to tell the linker "please put all of these together". The symbol then points to the start of this (now) array. I'm…not sure how ordering works or how it knows where "the end" is without looking into the implementation.
If the compiler is able to build a dependency graph, then the ordering would simply be a topological sort of the graph. If no such sort is possible, then there is a cycle, and the compiler can spit out an error. However, I have no idea if that's what would actually happen.
@CAD97, if you get a chance would you be willing to explain how this mechanism works a little more? Web searches turn up entirely unrelated topics, and although I'm searching through the compiler source right now, I suspect that reading the code will take far longer and be far less illuminating than your explaining it to us.
EDIT
My script just finished, distributed_slice does not appear to be mentioned anywhere in the rust sources, at least up to commit 17dfae79bbc3dabe1427073086acf7f7bd45148c.