Pre-RFC: Stabilize a version of the `rlib` format

[pcwalton]

Hi everyone! I've finally finished a pre-RFC for a solution to the problem raised in issue #73632. Thanks to everyone listed in the acknowledgements for very helpful feedback. Feel free to comment with your thoughts on this proposal!

Edit: I've posted version 2, incorporating all your helpful feedback!

Edit 2: Version 3 is up! It introduces the notion of standard library bundles, which specifies a full standardized solution for linking Rust binaries using the native Rust linker.

RFC: Stabilize a version of the rlib format

Patrick Walton pcwalton@fb.com

Pre-RFC version 3

Summary

Large projects written in multiple languages need to be able to link together multiple Rust crates that form complex dependency trees, each compiled with separate invocations of the Rust compiler. The current staticlib format exports symbols from dependencies of the crate being compiled, which can cause multiple-definition errors at link time. This RFC specifies an opt-in stable rlib format that external build systems can use to produce a library including only symbols from the crate being compiled, and no others, avoiding possible link errors.

Motivation

Frequently, Rust code is just one part of a large binary or dynamic library, perhaps built with a language-neutral build system other than Cargo, such as Bazel and Buck. In these projects, there may be arbitrary combinations of Rust and C++ code such that the same crate arises as a dependency at multiple points in the graph. The amount of investment in the toolchain and workflow for these projects frequently predates the introduction of Rust by years. Thus it is desirable to preserve a standard linking setup, in which the build system directly invokes the system linker (e.g. ld), in order to build a binary containing Rust code alongside code written in other languages.

Right now, the documented way to achieve this is by compiling the crate with the --crate-type=staticlib switch (or crate-type = ["staticlib"] in Cargo.toml). This works well for small projects. However, it has the fundamental problem that dependencies of the Rust crate being compiled are included in the resulting native library. This causes problems with diamond dependencies. Suppose that we have the following dependency hierarchy specified in the native build system:

Rust crates B and C, both compiled with staticlib, depend on the Rust crate A, while the C++ target D depends on B and C. Because of the semantics of staticlib, the contents of A will be duplicated into B and C. This can cause D to fail to link, because the linker can see definitions from A twice and exit with a "multiple definition" error. (Note that multiple definition errors are not guaranteed in the above scenario, because linkers are "lazy" and will only bring in symbols as requested. The success of the link is determined by the particular symbols in use in these four targets, as well as the number and makeup of each package's codegen units.)

The simplest way to solve this problem is to provide a supported way for the Rust compiler to produce artifacts that export only the symbols from the crate being compiled. That way, the build system, which has complete knowledge of the dependency graph, can produce a final link line that guarantees each crate is included only once in the resulting binary. In fact, Rust has a mechanism that is nearly perfect for handling this already (and which Cargo uses to solve this exact problem): the rlib format, which does not include symbols from dependent crates. However, the contents of rlibs are unstable, so external build systems can't technically use them without depending on implementation details of the compiler.

This RFC proposes an opt-in mechanism that external build systems can use to produce rlib files with a stable format. It's intentionally minimal and avoids stabilizing any more than is absolutely necessary for external build systems to work properly. Additionally, this RFC proposes a simple mechanism for packaging foundational Rust libraries such as the standard library into a format that the system linker can link against, allowing the creation of usable Rust binaries without rustc driving the linker.

Guide-level explanation

Compiler switch

A new compilation switch, -C rlib-version, is added to the compiler to control the contents of .rlib archives. It takes one of two values, with more possible in future versions of Rust:

• -C rlib-version=unstable — The default value, this option indicates that the contents of .rlib archives are unspecified. External tools should not rely on .rlib files conforming to any particular format.

• -C rlib-version=v0 — This value indicates that .rlib files conform to the version 0 format defined here.

Version 0 rlibs

A version 0 rlib is an archive file in the native format of the target, with the usual extension (.lib or .a) replaced by .rlib. The native format is the usual file format for statically-linked libraries on the target, which for all targets is some variation of the common ar archive format. (The format of WebAssembly rlibs is unspecified in this RFC.)

Inside the rlib file, any number of object files may be present that provide code and data for symbols defined by the Rust crate being compiled. Other files may also be present, such as .rmeta files. This RFC makes no guarantees whatsoever about what these files may or may not contain: in particular, this RFC doesn't stabilize any kind of metadata format. External tools such as linkers should ignore any non-object files, as their contents are unstable.

There must be a file inside the archive whose name begins with the string _rlib_v00. The contents are typically empty. The name of this file allows tools to determine the version of the rlib.

The object files inside a version 0 rlib must collectively contain global definitions for all the non-generic functions and statics defined by the crate being compiled. Global definitions must not be provided for any upstream dependencies of the crate, to avoid symbol collisions when linking. It's OK for functions for upstream dependencies to be present, but such symbols must be marked local to the archive. Symbol names should be appropriately mangled; in the case of v0 symbol mangling, they should follow Rust RFC 2603.

rlib files often contain undefined symbols with definitions in other objects, whether those objects be rlib files (i.e. crate dependencies) or other libraries such as native static or dynamic libraries. This RFC intentionally doesn't provide a way for an external tool to locate those dependencies. That's assumed to be the job of the build system.

These requirements are designed to allow non-rustc linkers to link executables created by the Rust compiler, driven by a variety of build systems, in a way that doesn't result in symbol conflicts when diamond dependencies are involved.

Standard library bundles

In order to successfully produce an binary containing both Rust code and native code, a way to link to the Rust standard library is needed. This RFC specifies a simple mechanism for doing so: simply compile an empty crate (an empty lib.rs file is fine) as a staticlib with a flag -C emit-std-bundle=yes. Any desired crate-level metadata and/or compiler flags can be supplied in the process of compiling this standard library bundle, for example #![no_std] to omit the standard library, or -C target-feature to enable specific CPU features. The resulting artifact will be a linkable version of the standard library.

An example workflow is as follows:

$ cargo new --lib stdrust
$ cd stdrust
$ echo "[lib]" >>Cargo.toml
$ echo "crate-type = ['staticlib']" >>Cargo.toml
$ RUSTFLAGS="-C emit-std-bundle=yes" cargo build --release
$ ls -l target/release/libstdrust.a
-rw-------  1 pcwalton  staff  17031504 Jan 19 19:37 target/release/libstdrust.a

The resulting libstdrust.a may be installed into the library search path, at which point -lstdrust may be added to the link line in order to link Rust executables.

Reference-level explanation

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119. They are not capitalized, for clarity.

rlib versioning

When the compiler is instructed to produce rlib output, the contents of the resulting artifact depend on the rlib version in use. The rlib version is specified by a compiler switch with the syntax -C rlib-version=VERSION, with VERSION replaced by one of the following:

• unstable — When the rlib version is unstable, the contents of the rlib file are completely unspecified by this RFC. In particular, the resulting rlib files may, or may not, actually be in v0 format. External tools should not assume that rlibs with version unstable conform to any specific format.

Note (non-normative): Most likely, unstable will result in a version 0 rlib being produced initially. The primary reason why unstable is left unspecified is so as not to preclude the possibility of MIR-only rlibs in the future.

• v0 — When the rlib version is v0, the contents of the rlib must match the definition supplied in the following section.

Other valid values of VERSION may exist. Their semantics are unspecified by this RFC.

Version 0 rlib contents

A version 0 rlib must be an archive file in the native format of the target. All supported targets use some variant of the common ar archive format. In particular, all supported targets begin their archive format with the string !<arch> followed by a newline character: i.e. the bytes 0x21 0x3C 0x61 0x72 0x63 0x68 0x3E 0x0A. The precise on-disk format of the archive file is unspecified by this RFC, but it must contain linkable object files as well as a symbol table.

For targets that do not use a variant of the common ar archive format, as well as WebAssembly, this RFC does not define the format of a version 0 rlib. Such platforms may or may not support version 0 rlibs at all.

In this section, we make reference to concepts of the BFD library. This provides a convenient way to abstract over the concepts that correspond to one another in different object formats.

Note (non-normative): BSD, System V, and Windows use incompatible mechanisms for specifying symbol tables inside the ar format, so we must use an abstraction.

The target crate is the crate that the current invocation of the compiler is compiling and producing a rlib artifact for.

A global symbol is a symbol with the BFD BSF_GLOBAL flag set. An rlib library defines whatever global symbols are required to link, subject to the three conditions below.

A local symbol is a symbol with the BFD BSF_LOCAL flag set. A rlib library may contain any number of local symbols. Their names and contents are unspecified by this RFC.

This RFC does not define the contents of the set of global symbols exported by an rlib archive. Instead, it requires that some number of global symbols shall be exported such that the following three conditions are fulfilled:

1. If some Rust crate B depends on Rust crate A, and both A and B are in .rlib format, the .rlib files corresponding to A and B shall be successfully linkable using the system linker, notwithstanding the requirements specified in the "additional linking requirements" section.

2. Any set of crates in .rlib format compiled by the same Rust compiler (including compiler version) must be linkable together as long as the following conditions are fulfilled:

a. For each crate, all dependencies of that crate must be in the set.

b. All conditions specified in the "additional linking requirements" section are met.

c. The set contains each .rlib file no more than once.

There are two exceptions to this rule:

(i) Multiple crates that define the same language item may not be linkable together.

(ii) Multiple crates that define identically-named items marked with #[no_mangle] may not be linkable together.

3. For each item with a #[no_mangle] annotation, a global symbol must be present in the archive with a name matching that of an identically-named C symbol definition on the target.

Note (non-normative): Some binary formats mark C symbols in some way (e.g. Mach-O represents them with a leading _).

Note (non-normative): "Linkable using the system linker" implies that there are neither undefined nor multiply-defined symbols.

Note (non-normative): Rust RFC 2603 specifies a mangling scheme for symbols.

Note (non-normative): Symbols relating to global allocation and panic handling must not be defined in the .rlib unless the crate itself defines those symbols.

The object files containing the symbols that the target crate defines shall be present inside the rlib archive. Any other files necessary for the Rust compiler to link to the target crate and use it as a dependency must also be present. The object files must be linkable in a format that the target supports.

Note (non-normative): Examples of linkable object files on various platforms include but are not limited to ELF, Mach-O, PE/COFF, LLVM bitcode for full LTO, LLVM bitcode for ThinLTO, and LLVM bitcode wrapped in a native object file.

Note (non-normative): The most important non-object file is the .rmeta file, which contains data necessary for downstream Rust invocations to use the crate as a dependency, such as the types and contents of inlined functions. The system linker ignores this information.

Additionally, a file with a name beginning with the string _rlib_v00 must be present inside the rlib archive. The contents of this file are unspecified. It allows build systems and other tools to determine that the rlib is in version 0 format.

Any other files may also be present inside the rlib archive. Whether these files exist, and what they contain, is unspecified by this RFC.

All rlib files that are to be linked together must be built in a compatible manner. The precise definition of compatible manner is unspecified by this RFC.

Note (non-normative): The reason for not defining the term compatible manner precisely is so that new linkage restrictions may be added in the future. For example, one could imagine a later version of Rust introducing multiple ABIs such as those suggested in the interoperable_abi proposal. In this case, the ABI of the rlibs and the ABI of the standard library bundle would all need to match for the link to succeed. This RFC is intended to preserve maximum flexibility for such changes in the future.

Standard library bundles

In order to successfully perform the final link of an executable containing Rust code, the core library must be present on the link line. Frequently, the std library must be present as well. This RFC specifies a mechanism to produce a standard library bundle containing core or std, appropriately configured to match the given target.

A crate successfully compiled as staticlib that contains no Rust symbols and no dependencies other than core or std with the -C emit-std-bundle=yes flag is known as a standard library bundle. The native system linker must be able to successfully link an executable containing Rust code if:

• All such crates containing Rust code are supplied precisely once to the system linker.

• All transitive rlib dependencies of all such crates are supplied precisely once to the system linker.

• Exactly one of the core or std standard library bundles is supplied to the system linker. This standard library bundle must have been built in a compatible manner with all rlibs to be linked.

Note (non-normative): At the time of writing, the -C emit-std-bundle=yes flag can simply be a no-op, as the Rust compiler can successfully create such staticlibs already by compiling an empty crate. The purpose of the flag is to ensure that this behavior is preserved in the future in an opt-in fashion.

The exact symbols that are exposed in a standard library bundle is unspecified by this RFC. In general, they are expected to change with every Rust release and may change depending on the manner in which the standard library channel was compiled.

Note (non-normative): The standard library bundle approach allows this RFC to avoid specifying details like the behavior of allocator shims, raw-dylib, bundled static libraries, #[global_allocator], the allocation error handler, -C panic=abort and -C panic=unwind, and so forth.

Drawbacks

This scheme doesn't preclude Rust changing the rlib format (for example, introducing MIR-only rlibs), but if Rust does so, under this RFC the compiler will need to retain support for the version 0 rlib support described here behind a compiler switch. This may add some amount of maintenance burden.

Addressing potential issues

• rustc now supports the concept of bundled static libraries, which are native libraries placed inside a .rlib file. The v0 rlib format doesn't support such libraries; generally, native build systems would prefer to keep libraries separate, for better interoperability with native code. This can be revisited with future rlib versions if need be.

• An issue regarding static initializers was raised during the discussion: they don't reliably work unless --whole-archive is provided when linking the rlib. However, --whole-archive is not available on AIX. AIX is currently not a supported platform for Rust, however; if and when it becomes one, the RFC for support for that platform can specify what to do here. Additionally, this is an issue that would be present regardless of whether the rlib format is specified.

Alternatives

• Keep the format of .rlib files unstable officially, but have external build systems depend on their format anyway. This wouldn't immediately have any ill effects, as external build systems like Buck and Bazel could depend on the contents of .rlib files and things would probably continue to work for some time. It would also have the advantage of avoiding the complexity of extra compiler switches and would allow the compiler to make a clean switch to MIR-only rlibs someday. However, this would cause breakage if Rust ever decides to change the format of rlibs.

• Have external build systems invoke rustc instead of ld to perform the final link. This would allow Rust to make a clean break with the past if it switches to MIR-only rlibs. It would also potentially obviate the need for the build system to be aware of the dependency graph, including standard library crates. However, it would force large C++ projects to switch linkers whenever they link in any Rust at all, which would significantly reduce the willingness of many C++ projects to incrementally adopt Rust by burdening the build system with extra logic. It would also be incompatible with any other language wanting to "take over" linking in this way; only one language can be in charge of the last linking stage, and the advantage of system ld is that it's language-neutral.

• Have external build systems invoke rustc to bundle all Rust dependencies together into one library, which is then linked into the final binary. This is similar to the previous alternative, except it adds an extra step. It would have the advantage of allowing rustc to automatically add extra libraries that need to be added to the final link line, such as allocator shims and native bundled libraries, without having to duplicate that logic into the external build system. However, this has potential performance issues due to needing to process Rust code twice, once with the rustc-invoked linker and once with the native linker. Additionally, this would complicate the common task of introducing Rust components to two unrelated portions of a large binary by requiring the build system to track every binary to determine whether Rust is involved and adding an extra global linking step if so.

• Add a new crate type, staticlib-nobundle or similar, which works like staticlib but without marking symbols from dependent crates global. This would mean that the same crate cannot be officially used from both Rust and C++, despite being essentially identical. In projects that have both Rust and C++ upstream crates that depend on a single Rust downstream crate, this would result in duplicate symbol errors as both the rlib and the staticlib-nobundle would be linked into the final binary.

• Use --emit=obj instead of using staticlibs or rlibs. With this approach, there is no obvious place for the Rust compiler to emit metadata (.rmeta files). Without metadata, the crate would no longer be linkable from Rust, only from C or C++, meaning that a library meant to be used from both C/C++ and Rust would need to be built twice. Additionally, this forces the number of codegen units to 1, causing compilation performance problems. Finally, this would have the same problem as staticlib-nobundle in that if both Rust and C++ link to the same crate, duplicate symbol errors would result, as Rust would be linking to an rlib and C++ would be linking to an object file with the same symbols.

• Use --emit=obj, and add support for multiple codegen units when using --emit=obj to rustc by having the compiler generate the object files separately and then use ld -r to link them together. The -r (relocatable) switch to ld allows multiple .o files to be combined into another .o file that can then be further linked into a binary. Unfortunately, this was tried early on in Rust's development and it was discovered that ld -r is often poorly supported by OS toolchains on account of how seldom the feature is used. Furthermore, this inherits the same problems mentioned before regarding metadata and duplicate symbols.

• Use an flag that doesn't carry a version number, like -C rlib-format=platform. This would be essentially the same as this RFC, but would not leave room for different versions in the future. For example, platforms might introduce new library formats in the future, or we might want to add some extra information to the .rlib format consumable by outside tools. In these cases, the ability to release a v1 version and beyond would be useful.

• Instruct the linker to discard duplicate Rust symbols instead of emitting errors, and have external build systems use -C crate-type=staticlib. The COMDAT feature in the ELF format (exposed as linkonce_odr in LLVM) can be used for this. This is what C++ does to avoid duplicate symbol errors when different object files include expansions of the same template. This solution gets the job done in practice, but it means that static libraries duplicate their dependencies, which results in extra needless I/O during the compilation (quadratic blow-up in the worst case). Moreover, it's inelegant.

• Stabilize the contents of .rlib files in perpetuity. This would prevent Rust from adopting MIR-only rlibs in the future, which are a commonly-discussed feature. The goal of this RFC isn't to hinder experimentation with alternative rlib formats.

References

See GitHub issue #73632 and the pre-RFC Discourse thread for the discussion that led up to this RFC.

Acknowledgements

Thanks to Jeremy Fitzhardinge, Matt Hammerly, Dana Jansens, Augie Fackler, Marcel Hlopko, and bjorn3 for feedback on this RFC, and everyone who took part in GitHub issue #73632 and the Discourse thread.

[CAD97]

Minor note: it makes some amount of sense to me to request this with a flag -C rlib-format=platform, and specify it as being the platform linker format with extra sections. This is, as far as I understand, just a different way of putting what you've described here.

There's probably something that I'm missing that makes that simpler definition insufficient, but it's probably worth it to list this as an alternative. (That flag also makes it obvious that it could support e.g. -C rlib-format=mir-only or -C rlib-format=proc-macro or any other number of named formats... with the downside of having to name the formats with something more than a version number.)

The RFC should probably also in the future-looking section mention if/how external build systems are expected to acquire a -Crlib-version=v0 version of std/alloc/core/sysroot for linking. If such a mechanism is provided, the note about import symbols could potentially be simplified to a link dependency on std/sysroot/etc.

---

That aside, I'm super happy to see some movement on this front.

[bjorn3]

Note that this pre-rfc supports neither raw-dylib nor bundled staticlibs and also doesn't allow getting the list of libraries that need to be linke whether native or rust libraries.

[josh]

It seems like this is specifying that an rmeta file or other secondary files will exist for rustc, but without providing any details for those files, and simultaneously saying (correctly) that the system linker ignores them. The combination of those statements does imply one critical property of the rmeta file and other secondary files: they're not allowed to have anything load-bearing in them that needs to be taken into account. Effectively, the rlib must be completely usable as a C library, while also being usable from Rust.

Also, as @bjorn3 noted, there's no mechanism here to get a list of what the staticlib would have linked in, which means that information would have to be duplicated into the other build system.

[pcwalton]

@bjorn3

Note that this pre-rfc supports neither raw-dylib nor bundled staticlibs and also doesn't allow getting the list of libraries that need to be linke whether native or rust libraries.

Dynamic linking can be handled in a followup RFC if there's a desire to do so. I don't think it makes sense to rush to specify dynamic linking semantics before we have concrete use cases, as it would be easy to get things wrong without understanding the problem space. The existing --crate-type=staticlib will still work and should handle the case of "bundled" static libraries.

@josh

The combination of those statements does imply one critical property of the rmeta file and other secondary files: they're not allowed to have anything load-bearing in them that needs to be taken into account.

rmeta files are allowed to have information that needs to be taken into account when linking from Rust, but not to have information that needs to be taken into account when linking from C/C++. System linkers generally have no concept of "metadata" beyond the symbol table and ar pseudo-filesystem, so there would be nothing to specify even if we wanted to.

@josh

Also, as @bjorn3 noted, there's no mechanism here to get a list of what the staticlib would have linked in, which means that information would have to be duplicated into the other build system.

That's correct and addressed in the RFC under "Addressing potential issues". I don't see any way around this: build systems want to pre-declare the dependencies up front instead of, for example, querying rustc for that information on the fly. Invoking rustc to do the final link is not really a viable option for the reasons outlined in the Alternatives section.

[bjorn3]

@pcwalton

Dynamic linking can be handled in a followup RFC if there's a desire to do so. I don't think it makes sense to rush to specify dynamic linking semantics before we have concrete use cases, as it would be easy to get things wrong without understanding the problem space. The existing --crate-type=staticlib will still work and should handle the case of "bundled" static libraries.

raw-dylib is not a crate type, but a link type. It allows linking to existing libraries like libc without requiring those dynamic libraries to be present on the build system. Libstd will likely use it on Windows in the future to make cross compilation easier. The way raw-dylib works is by creating an import library that needs to be passed to the linker. As for bundled static libraries, that isn't a crate type either. Instead it is that static libraried linked into an rlib are directly bundled into the rlib. This is used by compiler-builtins and on some targets by the libc crate. It works by embedding the staticlib archive as member of the rlib archive. Both features are or will be necessary for supporting libstd or even #![no_std].

[farnz]

@pcwalton

Have external build systems invoke rustc instead of ld to perform the final link. This would allow Rust to make a clean break with the past if it switches to MIR-only rlibs. It would also potentially obviate the need for the build system to be aware of the dependency graph, including standard library crates. However, it would force large C++ projects to switch linkers whenever they link in any Rust at all, which would significantly reduce the willingness of many C++ projects to incrementally adopt Rust by burdening the build system with extra logic. It would also be incompatible with any other language wanting to "take over" linking in this way; only one language can be in charge of the last linking stage, and the advantage of system ld is that it's language-neutral.

A follow-up question on this one: would it be workable to have an invocation of rustc that takes all the .rlib files and generates a final system library for the build system to link in?

In other words, a special rustc invocation that's the last Rust build step, linking all the Rust dependencies together, and producing something for the system linker to take on?

[bjorn3]

That is already possible. Just generate a source file containing use liba; use libb; use libc; ... and then compile it as staticlib. This is bothersome in build systems like bazel as it requires knowing all rust libraries that will end up getting linked in central location. It also doesn't help for rust for linux as it is incompatible with dynamic linking of extra code against this bundle, as would be necessary for kernel modules.

[lu_zero]

Without having a (better) mean to provide the link-line needed when you add external libraries, only proposing a standardized archive with less symbols seems to not solve the foreign build-system problem.

Probably you should expand on how the proposed rlib-v0 is different from staticlib-nobundle.

Are the ld -r problems tracked somewhere and the respective upstreams made aware? It seems a much neater alternative.

[bjorn3]

@lu_zero

Are the ld -r problems tracked somewhere and the respective upstreams made aware? It seems a much neater alternative.

On macOS it seems to not export any symbols. In any case you can already use --emit obj to emit a single object file.

[bjorn3]

@pcwalton

In order to successfully link a Rust target, a global allocator crate must be present, in order to provide definitions of symbols relating to memory allocation. This RFC does not define the ABI that the global allocator crate exposes and that the compiler relies on; it only specifies that some ABI exists.

Global allocators aren't the only issue when using libstd. There are several symbols for handling allocator issues that are only emitted by the global allocator shim even with my PR to allow #[global_allocator] without allocator shim. They are mostly concerned with allocation failure. Your rlib-v0 proposal doesn't solve this afaict.

[pcwalton]

@lu_zero

Without having a (better) mean to provide the link-line needed when you add external libraries, only proposing a standardized archive with less symbols seems to not solve the foreign build-system problem.

I mentioned this in the RFC. If we need a standard solution for determining the set of libraries needed to link against Rust, then that can be a followup RFC, as that's a separate problem. As is, this RFC is an improvement on the status quo that solves the most pressing issue.

@lu_zero

Are the ld -r problems tracked somewhere and the respective upstreams made aware? It seems a much neater alternative.

As the RFC states, ld -r inherits the same problem as staticlib-nobundle:

This would mean that the same crate cannot be officially used from both Rust and C++, despite being essentially identical. In projects that have both Rust and C++ upstream crates that depend on a single Rust downstream crate, this would result in duplicate symbol errors as both the rlib and the staticlib-nobundle would be linked into the final binary.

I don't see any solution for this, even if the other problems were fixed; it seems fatal to ld -r.

[pcwalton]

@bjorn3

Global allocators aren't the only issue when using libstd. There are several symbols for handling allocator issues that are only emitted by the global allocator shim even with my PR to allow #[global_allocator] without allocator shim. They are mostly concerned with allocation failure. Your rlib-v0 proposal doesn't solve this afaict.

Thanks for the feedback! I'll look into what's needed to address this in the next revision of the RFC.

[keith]

@pcwalton

However, it would force large C++ projects to switch linkers whenever they link in any Rust at all, which would significantly reduce the willingness of many C++ projects to incrementally adopt Rust by burdening the build system with extra logic.

As another data point against this alternative, when using languages besides C++, like Swift on iOS, with rust, the competition for "who gets to do the final link" is even higher cost than "are you willing to switch linkers", as other communities may be less likely to implement a RFC similar to this and therefore require their executables to drive the link.

@pcwalton

With this approach, there is no obvious place for the Rust compiler to emit metadata (.rmeta files).

The approach that Swift takes in a similar case is to embed this info into a standard object file in a section that is ignored by other tools. This might have been discussed previously in rust's path to using separate .rmeta files, but I think it's an interesting data point.

[bjorn3]

We already wrap the crate metadata in an object file on most targets, but when the object file format is not supported by the object crate (for example wasm) we don't wrap it.

[jsgf]

@farnz

A follow-up question on this one: would it be workable to have an invocation of rustc that takes all the .rlib files and generates a final system library for the build system to link in?

In other words, a special rustc invocation that's the last Rust build step, linking all the Rust dependencies together, and producing something for the system linker to take on?

If we assume that Rust and non-Rust can be freely intermixed throughout the build dependency graph, then it means that the only place where that pre-link step could happen is just before the final link when all the object files are available. In practice this means it would be a wrapper around the linker which is akin to having rustc itself do the final link. (I guess it might help things if the final linker is doing complex things like Haskell (or Swift, I gather).)

But I think it would be better to avoid it if possible. A key goal here is the ability to integrate Rust into an existing build system while localizing the knowledge that any particular component is written in Rust. That is, so you can RIIR a module without having to worry about how all the downstream users of that module are built.

[simonbuchan]

I might be missing something obvious, not being an expert at all here, but does this compare or interact with generic instantiations?

Eg in the given diamond diagram, if both B and C instantiate an identical type from A, eg Vec<AType> or ACollection<String>, then even Rust-only (and C++ only, etc...) code can end up with duplicate symbols. My understanding is that since they're mangled, it's considered perfectly ok to tell the linker to just pick one - they have to refer to the same code anyway.

So, my question is, since this is the same basic problem as you have here with all of A's code, is there any reason the same solution doesn't work here? And if there is, why isn't it a problem with instantiated generics?

My best guess is that this on Unix-likes is implemented with weak symbols, which also have a bunch of other semantics like not being an error if they have no definition and so on, which isn't what you want for most symbols. With Rust's semantics, in particular mangling, is this still a problem? Eg you can't do the equivalent of C/++ global int undefined_symbol; (with no extern, static or initializer) to declare but not initialize a symbol, which it seems from some investigation creates a half-defined symbol that can merge with a fully defined variable, but doesn't cause a missing symbol error if there is no other.

[scentini]

A follow-up question on this one: would it be workable to have an invocation of rustc that takes all the .rlib files and generates a final system library for the build system to link in?

I see two downsides of using a pre-link rustc invocation (in a mixed Rust and non-Rust codebase), which might make it prohibitively expensive:

• The pre-link step is on the critical path of every non-Rust executable, increasing the total build time.
• Memory footprint in the build cache: Any non-Rust test (and all other executables) that contains Rust in the dependency tree will need to link its Rust dependencies into a staticlib. In a mixed codebase with non-restricted Rust usage, this means having many different staticlibs (up to one per non-Rust executable). Each staticlib contains its transitive Rust dependency tree, so it can be quite big in size; thus, caching the pre-link artifacts will result in a significant memory increase.

[tschuett]

As an outsider: Could you teach libObject in LLVM to read rlib files and teach the lld linker in LLVM to link rlib files?

[bjorn3]

@simonbuchan

I might be missing something obvious, not being an expert at all here, but does this compare or interact with generic instantiations?

Generics functions are never exported, so there is no chance of symbol conflicts.