Pre-RFC: Stabilize a version of the `rlib` format

The issue is not that lld or other linkers can't read rlib files in their current implementation (they are standard ar archives), but that this isn't guaranteed to be the case and that there are object files and link args essential for linking that rustc only generates when it invokes the linker by itself. Without these object files liballoc and libstd simply fail to link due to missing symbols and without the link args we may be missing essential libraries or in the past fail to link due to cyclic dependencies (we solved this by producing a symbols.o file referencing all symbols we need in advance, but that file is again only produced right before rustc invokes the linker)

Yep, that would do it!

One purly API related alternative could also consider is, rather them adding a new compilation flag, rlib-version, a new crate type, e.g. (staticlib-nobundle or object-rlib) is added that has the semantics of the v0-rlibs described in this proposal (including the ability to be linkable into other rust code.). This would avoid having to add an API switch, only usefull for one single crate type. Also this would allow for a future use case where users want to generate a MIR rlib first and the build an v0-rlib from it.

Unrelated to this: Would there be any advantages of somehow matching up the layout to C++ compiled modules?

So, I'm going to preface this by saying that I am in favor of the general idea of a compatibility initiative, however there are some problems with just stablizing the rlib format.

As part of the lccc project, I have begun specifying a considerable portion of the associated formats (including rlibs and the rmanifest file) and abi details, and considerable is indeed the proper adjective here. To be useful, beyond for inspection by curious programmers (which far from requires a specified format), many portions of what are currently internal compiler details must be made public, at the very least on a by-version basis.

This definition does not address what must be a global symbol. Do #[inline] definitions need to be global? And of course, if they are, do they only need to be defined in the object that includes them (this falls short, as by design, they are generated in multiple cgus to be inlined, and are compiled as weak symbols that are presumably placed into a linkonce COMDAT group). What about (instantiated) generic symbols. What about destructors (the drop_in_place definition, not the Drop::drop function specifically). Additional shims? I think you'd find an exhaustive list to be nearly impossible to specify without nailing down many of the abi details. Even with those details, the lcrust abi lacks any sort of exhaustive list, and only has a few cases that it says "must be generated" (implying the symbol is global). It's entirely possible that a symbol could be "public" but not "global" and require rustc to make it available when linking downstream. This specification does not prevent that, and I would assume it would require a full ABI specification to do so.

It's entirely possible that an .rmeta file may be an object file with the metadata in a section, and could contain salient definitions (including symbols containing no data, but are checked by code in other object files). This would preclude doing that.

Are you talking about C++ standard modules? Or are you referring to compiled TU objects that get put into some container ("library")? All of these are platform specific (the former is even compiler-specific) and wholly unspecified in any language spec.

1 Like

C++20 standard modules as implemented by e.g. clang. This was just a wilde guess, I don't know if there would be any benefit at all.

No, I don't think there'd be any benefit (as an ISO C++ committee member on SG15 and CMake developer working on C++20 modules support).

As far as I understand it C++20 modules as implemented by clang and gcc are basically dumps of the internal compiler structures of the respective C++ frontend. This format is fundamentally incompatible with most non-C++ languages, differs between clang and gcc and does not capture any of the complexity that makes it hard/impossible for rlibs to be linked without rustc wrapping the linker.

1 Like

How does that differ from this proposal in practice? Other than trying to convince all the existing Rust tooling to emit a different --crate-type option to rustc?

(I think a related issue is that the roles of the --crate-type and --emit options are pretty confused and entangled, and could do with a solid bout of rationalization.)

This would avoid having to add an API switch, only usefull for one single crate type.

I'm not sure what you mean by this - specifically how "API" comes into this, beyond the very general sense of an object file being part of the "API" of a toolchain.

There are three mutually exclusive crate type families: proc-macro, bin and the various library types. Selecting one family will have affects at the frontend, unlike --emit. --emit affects the backend with the various library crate types kind of being sub types of --emit link. Now this is not an entirely accurate view as which --emit and which library types are used can affect what the crate metadata contains and the library types have an effect on the crate metadata loading as well as some other things, but I think it is a usable mental model despite these limitations.

1 Like

To make the stable rlib format useful you did have to have the standard library available in this format too. Cargo doesn't specifying the crate types of crates on the commandline as would be necessary for this. It hard codes them in Cargo.toml.

I'm not sure what you mean by this - specifically how "API" comes into this, beyond the very general sense of an object file being part of the "API" of a toolchain.

API is ment in the sense of "the choice of available command line arguments of rustc".

What I mean here is that the suggested API in the pre-RFC, suggests that -C rlib-version=v0 is somehow orthogonal to crate type, e.g. that something like: -crate-type=staticlib -C rlib-version=v0 could be somehow usefull. However choosing to specify rlib-version, somehow implies, -crate-type=rlib. My suggestion removes this "redundancy" by spliting the crate type rather them adding a new switch.

How does that differ from this proposal in practice? Other than trying to convince all the existing Rust tooling to emit a different --crate-type option to rustc?

There should be little to no difference in practice, this is mostly an command API layout proposal. The effect on tools will also be similar, in both cases tools would need to be ajusted to make use of the stablized format.

I don't think so. I would expect that you can use RUSTFLAGS="-Crlib-version=v0" and then have all rlibs use this version, but have all non-rlib crate be unaffected.

1 Like

So I've been thinking about this, and I think we can get away without specifying anything beyond the bare minimum of API details by doing the following, and no more:

  1. Specify that, if any Rust crate B depends on Rust crate A, A must export whatever symbols are necessary for B to successfully link. We don't need to specify what those symbols are, or even whether they're weak or strong; that's an implementation detail that can and will change from rustc version to rustc version. We just need to say that enough symbols are exported, whatever those are, to avoid missing symbol errors in the final link.

  2. Specify that if any Rust crate C depends on Rust crate A, there will never be duplicate symbol errors in the final link, as long as there is only one copy of C and A in the link line.

  3. Specify that all pub extern functions are exported, under their mangled name (deferring to the name mangling RFC) or under their unmangled name if appropriate. This ensures that C++ code can link to Rust code as necessary.

This spec language is a trick that lets us avoid specifying ABIs.

It's entirely possible that an .rmeta file may be an object file with the metadata in a section, and could contain salient definitions (including symbols containing no data, but are checked by code in other object files). This would preclude doing that.

Can you elaborate as to what "checked by code in other object files" means? For any object files A and B (in this case B would be the .rmeta file), either A contains a symbol reference to B or it does not. If it does, then the linker will detect that and will find the object file for linking. If it doesn't, then there should be no problems.

1 Like

I've posted version 2 at the top of this thread, incorporating everyone's helpful feedback. Thank you!

1 Like

I think only #[no_mangle] functions should be guaranteed to be exported. No function with a mangled name can be called anyway by non-rust code as the mangled name contains an unpredictable hash (both for the legacy and v0 mangling scheme). Furthermore cdylib only exports #[no_mangle] functions anyway.

Several linkers entirely forbid non-object files. That is the reason we wrap the crate metadata in an object file.

#[inline] functions are never exported from object files.

In that case should we disallow mixing multiple rlib versions in the same crate graph?

AIX begins it with <bigaf>. By the way should we guarantee a specific archive format if multiple are supported on a target? On almost all platforms we default to the gnu archive format (32/64bit variant depending on archive size), even on bsd targets that have their own archive format. Pretty much only macOS and AIX would use a different archive format.

This misses #[global_allocator] and the alloc error handler. In addition it misses the rule that two libraries with identical crate name need to have different -Cmetadata arguments to disambiguate symbols.

Some internal unmangled symbols don't start with __. For example rust_eh_personality. Not sure if any such symbols are generated by the compiler though.

It isn't an issue for rustc as it generates a symbols.o file containing references to all symbols exported by any rlib (with the exception of those defined by bundled static libs) to force all object files in all rlibs to be linked.

What if rustc were to have a mode in which it becomes a transparent wrapper around whichever linker except for also handling rust rlibs such that you can just squeeze it in between the other language and gcc/clang? Then it did just be a matter of stacking the linker drivers for all involved languages on top of each other, right? Also the gcc/clang linker drivers are not language neutral. They contain a lot of code to handle C/C++ peculiarities (like determining which system libraries or compiler specific libraries (like libgcc_s or libcompiler_rt) to link or handling static initializers) before ever running the actual ld linker. This is why rustc uses gcc/clang rather than the linker directly.

By the way how should the standard library be handled? It can't be built on stable, yet it did need to use the v0 rlib version and the build system needs some way to get the location of standard library crates and their dependency lists.

Next stupid question: Clang has the concept of toolchains, i.e., Darwin, Windows, some BSD, and PS5. If you would add a Rust toolchain, could it find the missing pieces before invoking lld?

The code for determining what libraries to link and with which linker arguments and what to object files generate is specific to each rustc version and generating those object files literally runs the codegen backend. However we need to work with gcc and clang versions that are both older and newer than rustc. Additionally I would expect toolchains to be mutually exclusive in clang, but rustc would need to be additive as you can still link C code. And finally gcc doesn't have this concept of runtime switchable toolchains.

This misses #[global_allocator] and the alloc error handler.

I recall that there have been discussions about generalizing that pattern to a sort of provided-impl mechanism where a crate can require that a type implementing some trait must exist and will be provided exactly once in the compilation graph, usually but not necessarily by the final bin crate.

Those would probably run into similar issues?