Meaning of #[link] kinds

language design
1

As I’m sure most people here know, the #[link] attribute used to link to native libraries for FFI has a kind option, which can take one of three values:

  • kind = "dylib" (the default)
  • kind = "static"
  • kind = "framework" (OS X specific)

I’ve been experimenting on Linux, so I’m going to ignore framework for the rest of this post.

The FFI guide describes the meanings of these options, and at least implies that dylibs should be linked against using the “dylib” kind (or more commonly by leaving it out) and static libraries should be linked using the “static” kind, as one would expect from the names. This has been confirmed to me on the pull request that started me looking at this. (After that I experimented with having rustc determine what type of library it was linking and choose the link kind automatically, but it turns out that is a bad idea.)

However, that isn’t actually what the option does internally, nor is it how it is used in practice. At least one test in the rust repository links to static libraries without specifying a kind, and when linking against system libraries one often does not know what form they are going to be provided in on a user’s computer. (This is causing distros trouble with using external LLVM.)

The FFI guide section mentioned above actually does accurately describe what the option does: when compiling an rlib, a “static” library is bundled into the generated file, while a “dylib” library is just recorded as a dependency and not linked until an executable (or dylib) using the rlib is created. This does mean that a “static” library must be an actual static library, because one can’t usefully put a dylib into an archive. The “dylib” kind will, however, work for either a dylib or a static library, and in either case will delay linking it in until creating an executable.

It turns out that this is actually the desired behavior for static libraries much of the time. If the test mentioned above is changed to “correctly” specify that it is linking against a static library, it fails to build with multiply defined symbol errors. In terms of today’s implementation, it seems that the meanings of these options are roughly

  • kind = "static" == bundled
  • kind = "dylib" == external (available on the system)

and linking against a system library with with the “static” link type will actually break things if it is done in both an executable crate and an rlib crate that it uses.

The current behavior actually seems more useful to me than what appears to have been intended, but it means the names are very confusing.

What I’m wondering is: Does this all seem correct? Do people think the current behavior is a bug? Should the option names or documentation be changed?

1 Like
2

This behavior is really confusing in my opinion. It caused me a lot of frustration and confusion in https://github.com/rust-lang/rust/issues/27438.

Note that on Windows there is no distinction between static and dynamic libraries. Everything is just a .lib library. In that library however those symbols themselves might be static or dynamic, dynamic symbols just redirecting to some .dll. What this also means is that you can use such a library with either kind=static or kind=dylib and it’ll work either way regardless of whether the symbols are static or dynamic. You just need to make sure dllimport is either applied or not applied depending upon whether the symbol is dynamic otherwise it’ll be inefficient or not work.

Therefore I think the terms static and dylib are very confusing based on what the actual behavior is, especially since on Windows there is zero correlation between static/dylib and a library having static or dynamic symbols.

Also in a Cargo based world I really don’t see the need for kind=static since the libraries are guaranteed to hang around until link time and deferring it to link time always works. Are there really ever any cases where static=kind's bundling is actually needed?

3

It sounds like a lot of your confusion may be derivative from the fact that the system linker (ld) doesn't get told "link this library statically" or "link this library dynamically" (it doesn't have an option to do that really), it just gets told "link to this library". This in turn ends up meaning that linking to a static library without actually saying it's static will work in some situations. This is not guaranteed to work in all situations, however.

It's also important to understand that the propagation behavior of linking to a native library is very important for the kind directive you're passing down to the compiler. Once a crate links to a dynamic native library the library is passed to the linker every time the crate is then reused downstream. A static library, however, is bundled inside of an rlib, so it's never passed to the linker again. This behavior is crucial to get native libraries working, and is why this distinction exists.

Many of these kinds of tests predate the existence of kind = "static" so not using it when you should is certainly not idiomatic, the test is wrong in this case.

I don't think I understand what you mean by this, can you elaborate? Does knowing that it's used to alter propagation behavior make sense?

What you've said seems correct to me, and it's all intentional (e.g. none of this is a bug), and I'd like to hear more about why you think that names should be changed!

4

Okay, I’m going to reason about how kind=static vs kind=dylib can be useful in the scope of the Windows world and DLLs.

Let’s say you have a static C library called foo. It has global state so you want to make sure there is only a single instance of it. The linker is invoked to create a binary each time a .exe or a .dll is produced. If you pass foo to the linker for a given binary, it will gain its own instance of foo. If you have a Rust library that uses foo built into a .dll, and then you use that .dll in a .exe, you really want to make sure there is only a single foo. Therefore foo should not be passed to the linker for the .exe and it should only be passed to the linker for the .dll containing the Rust library that links to foo. foo will in turn be exported from that .dll as needed. This is the situation where kind=static makes sense, but notice that bundling foo into the .rlib is not the solution, the real solution is simply making sure foo is only passed to a single linker invocation and re-exported for use by the other binaries.

Now suppose we have a C library which does delegate to a .dll. In this case you don’t have to worry about global state since it is already a .dll, therefore you can use kind=dylib and just happily pass it along to each linker invocation and everything will turn out just fine.

Maybe on non-windows bundling is the answer, but in the windows world, bundling solves absolutely nothing. The real solution is just making sure global state isn’t duplicated across multiple binaries.

5

(I haven't quite figured out how quoting works here yet. Sorry if this is a mess.)

I see that it is used to alter propagation, but I don't understand what that has to do with static vs. dynamic linkage. For example, the following simplified form of the test I mentioned previously, (which I think is valid and is using #[link] as intended) fails to build:

$ cat native.c 
#include <stdio.h>
void hello(void) { printf("Hello\n"); }
void world(void) { printf("World\n"); }
$ cat dependency.rs 
#![crate_type = "rlib"]

#[cfg_attr(not(dylib), link(name = "native", kind = "static"))]
#[cfg_attr(dylib, link(name = "native", kind = "dylib"))]
extern {
    fn hello();
}

pub fn say_hello() { unsafe { hello(); } }
$ cat executable.rs 
extern crate dependency;

#[cfg_attr(not(dylib), link(name = "native", kind = "static"))]
#[cfg_attr(dylib, link(name = "native", kind = "dylib"))]
extern {
    fn world();
}

fn main() {
    dependency::say_hello();
    unsafe { world(); }
}
$ gcc -fPIC native.c -c ; ar cru libnative.a native.o
$ rustc -L. dependency.rs 
$ rustc -L. executable.rs 
error: linking with `cc` failed: exit code: 1
note: "cc" "-Wl,--as-needed" "-m64" "-L" "/usr/lib64/rust-1.2.0/rustlib/x86_64-unknown-linux-gnu/lib" "executable.o" "-o" "executable" "-Wl,--whole-archive" "-l" "morestack" "-Wl,--no-whole-archive" "-Wl,--gc-sections" "-pie" "-nodefaultlibs" "/home/wthrowe/tmp/foo/libdependency.rlib" "/usr/lib64/rust-1.2.0/rustlib/x86_64-unknown-linux-gnu/lib/libstd-gentoo-stable.rlib" "/usr/lib64/rust-1.2.0/rustlib/x86_64-unknown-linux-gnu/lib/libcollections-gentoo-stable.rlib" "/usr/lib64/rust-1.2.0/rustlib/x86_64-unknown-linux-gnu/lib/librustc_unicode-gentoo-stable.rlib" "/usr/lib64/rust-1.2.0/rustlib/x86_64-unknown-linux-gnu/lib/librand-gentoo-stable.rlib" "/usr/lib64/rust-1.2.0/rustlib/x86_64-unknown-linux-gnu/lib/liballoc-gentoo-stable.rlib" "/usr/lib64/rust-1.2.0/rustlib/x86_64-unknown-linux-gnu/lib/liblibc-gentoo-stable.rlib" "/usr/lib64/rust-1.2.0/rustlib/x86_64-unknown-linux-gnu/lib/libcore-gentoo-stable.rlib" "-L" "." "-L" "/usr/lib64/rust-1.2.0/rustlib/x86_64-unknown-linux-gnu/lib" "-L" "/home/wthrowe/tmp/foo/.rust/lib64/rust-1.2.0/x86_64-unknown-linux-gnu" "-L" "/home/wthrowe/tmp/foo/lib64/rust-1.2.0/x86_64-unknown-linux-gnu" "-Wl,-Bstatic" "-Wl,--whole-archive" "-l" "native" "-Wl,--no-whole-archive" "-Wl,-Bdynamic" "-l" "dl" "-l" "pthread" "-l" "rt" "-l" "gcc_s" "-l" "pthread" "-l" "c" "-l" "m" "-l" "compiler-rt"
note: ./libnative.a(native.o): In function `hello':
native.c:(.text+0x0): multiple definition of `hello'
/home/wthrowe/tmp/foo/libdependency.rlib(r-native-native.o):native.c:(.text+0x0): first defined here
./libnative.a(native.o): In function `world':
native.c:(.text+0x12): multiple definition of `world'
/home/wthrowe/tmp/foo/libdependency.rlib(r-native-native.o):native.c:(.text+0x12): first defined here
collect2: error: ld returned 1 exit status

error: aborting due to previous error

but if I use kind = "dylib" for my static library, it works:

$ rustc -L. dependency.rs --cfg dylib
$ rustc -L. executable.rs --cfg dylib
$ ./executable 
Hello
World

Moreover, if I examine the actual underlying commands manipulating the object files, the "dylib" version does what I expect from a C perspective.

If it is intended that static libraries should only be linked with kind = "static" and dylibs should only be linked with kind = "dylib" then the names seem fine to me. I only think it is confusing if kind = "dylib" is intended to be used for static libraries as well, which it sounds like it is not.

@retep998 Looks like we came to the same conclusions. It's good to know that this also seems strange from a Windows perspective (an area that I know nothing about). (And it sounds like this actually manages to be more complicated there than on Linux. Ow.)

6

Ah yes your test case fails because you're statically linking the same library in two separate compilation units. This is invalid and is the cause of endless weirdness and crashes in the real world, and this happens to manifest itself as a link error in Rust. Static libraries are intended to be linked once and only once.

To be clear, there's nothing Windows-specific about the linkage behavior Rust does beyond differences in link.exe and ld. Treatment of native libraries from the compiler's perspective is the exact same across all platforms.

7

dllimport-ness is orthogonal to what #[link] controls - it is the Windows (inverse-sense) equivalent of visibility("protected"), except both modes are popular and LINK.EXE has that wart when it doesn’t generate __imp_ in some cases.

8

It’s kinda unfortunate that those library kinds were named “dylib” and “static”. If they were called “normal” and “bundled”, there’d be way less confusion. Especially now that we have MSVC linker in the mix, where “static” and “dynamic” linking mean something else.

9

So then, how would I write an executable that wants to use the same native library as one of its dependencies?

10

I disagree with this, "dylib" and "static" are exactly what's going on here. When you link against a dynamic library you have to keep linking against it to in order to resolve downstream references. When you link against a static library you bundle it once and you're done. It's incorrect to link to a static library more than once, and it's also incorrect to link to a dynamic library only once.

The MSVC linker doesn't add anything new here beyond the fact that both static and dynamic libraries are linked against via foo.lib, meaning you can't have both a static and a dynamic copy of the same library. Beyond that the behavior of linkage is the same where dylibs must be linked many times and static libraries must only be linked once.

The idiomatic (and possibly more correct) way to do this would be to have the FFI functions be reachable from the dependency, e.g :

// a.rs
#[link(name = "foo", kind = "static")]
extern {
    pub fn foo();
    pub fn bar();
}

// b.rs

extern crate a;

pub fn foo() {
    unsafe { 
        a::foo();
        a::bar();
    }
}

The less idiomatic method (and possibly less correct) would be to just copy the definitions without #[link]

11

I believe that this is not a hard rule. Some libraries tolerate multiple copies (even multiple versions) of itself in the same process just fine. At least on Windows. In fact, msvcrt is like that, as long as you are careful to not free resources allocated by one copy in another.

Edit: I will concede though that linking only once is the safe default behavior, which should be encouraged.

12

No, this is different from what it means to statically link in MSVC: the "bundle" behaviour that occurs with kind = static never happens in normal MSVC usage (although you can get it by manually invoking link.exe).

For example, the usual way to depend on a static library is to add this line to a header file:

#pragma comment(lib, "static_dependency.lib")

It's perfectly valid for this to occur multiple times, and when the header file is compiled as part of a static library, it will essentially be ignored (you can't specify linker inputs when building a static library).

However, when the header file is later used by a downstream .dll or .exe then static_dependency.lib will be added as a linker input. So static linking in MSVC behaves the same as kind = dylib in urst.

13

I was thinking of the case where the use of the library by the dependency crate is an implementation detail. It seems very unfortunate that, for example, changing a math library to internally use blas is a breaking API change (assuming libblas is linked staticaly, which is not uncommon).

Edit: And is also a breaking API change for all reverse dependencies of your crate.

Edit 2: I think what would help me the most here is an example of why this behavior is useful when linking against a system static library. (It makes sense to me for the internal C portion of one’s own crate.)

14

Really what I think dylib vs static should be used for is to differentiate between whether a library should only be linked once since it is static and has global state that shouldn’t be duplicated, or if it should be linked all the time because it is a dynamic library so state duplication isn’t a concern.

The fact that static causes bundling is a very unfortunate side effect for two reasons:

  1. On Windows it is entirely possible to bundle a library with dynamic symbols, or not bundle a library with static symbols, both in exact opposite to the pairing static dictates. All four combinations are legal in the Windows world, so why does Rust insist on only providing two of them?
  2. Since it is now rustc’s job to find the library, instead of the linker, that means the directories in which rustc searches for that library are significantly different. This is especially problematic with system libraries, like those in the Windows SDK, because I don’t know where those libraries are, and if I did want to tell rustc where to look I’d have to replicate all the logic of the linker for looking for those libraries.

Basically, there’s no way to link a static library such that it is only linked once but without incurring the bundling behavior and thus the different search paths.

15

@Diggsey it's important to keep in mind that we're pretty far removed from whatever the cl.exe compiler does as we're driving link.exe directly. Things like that #pragma I believe just emit directives into the object file being generated which are then later parsed and dealt with in the linker. This means that everything in MSVC translates to some invocation of the linker, which is what's relatable to what we're doing in Rust.

I'm not really sure what it means to use a header in a dll or exe, but it'd be useful, however, to analyze the full process there. The header is included by some file which is then translated to an object file which is in turn passed to the linker (with these embedded directives). This all translates to some set of libraries being passed to the linker which may or may not look similar to what we do in Rust.

A primary point of the static linkage in Rust is to make distribution of native dependencies possible without distributing the libraries themselves. For example this means that we can build small C shims into an rlib and then forget about them, we don't have to manually ship them everywhere next to our artifacts.

This is unfortunately simply a downside of C, you can't really have private implementation details in a global namespace where you're just using different versions of the same C library because both copies of the library have the same symbols.

Primarily this is quite rare, system libraries are almost never static libraries. This behavior is useful, however, to ensure that the linker robustly succeeds in all cases as well as everything gets resolved to the right invocation. Failure modes include:

  • If a library isn't bundled then an intermediate artifact which only included half of it could cause a linker failure somewhere else down the road.
  • If you're using two copies of a library, some library calls may be routed to one while others may be routed to another, this is almost always a situation where segfaults and unexpected behavior arises.
  • Distributing a static library without having to actually distribute the library itself is quite useful

I think you're conflating a lot of issues together here and it would be beneficial to tease them apart. It looks like this is quickly going down a slippery slope of "everything is awful" which makes it difficult to reason about what needs to be addressed vs what needs to be understood. I'll try to address each point of this in turn.

Can you be more concrete here? I don't understand what this looks like in Rust, what the failure modes are, or what we should do as a result.

I think this is a red herring, this is the exact same behavior for all platforms. The compiler does not know about the "system linker paths" and it's just unfortunate that we have to actually calculate it on Windows. In order to be consistent here we would have to do this across all platforms, not just Windows.

As a result, this seems quite orthogonal to anything having to do with #[link].

That's because this is incorrect. When a linker pulls in a static library it typically only pulls in the parts that are needed, not the entire library. This means that if you create a Rust dylib which links to a native static library it may not actually include all of the static library, which in turn can cause future linkage to fail.

16

Exactly, IMO this is the rust-specific concept of "bundling" and is conceptually unrelated to whether you statically or dynamically link a dependency. What you are defining as kind = dylib is exactly the process used to link both static and dynamic libraries on windows (there is no difference as far as the linker is concerned, thanks to import libraries).

It's clearly not entirely incorrect as this is the primary way libraries are statically linked (at least on windows). You have to remember that on windows .exes and .dlls are essentially the same thing, so the expected behaviour is that each dll/exe gets its own copy of any native static library dependencies, and so future linkages cannot fail (once you produce a dll or exe, there are no more "future linkages").

17

The linker will include anything which is dllexported or referenced from dllexported functions, regardless of how it was supplied to the linker. Whether it was bundled into the .rlib or provided as a separate .lib, link.exe doesn't really care (especially when called with /OPT:REF). If someone has a Rust library that wraps a static library, and the Rust library is built as a DLL, then Rust should apply dllexport to all symbols publicly visible from the DLL. If a symbol in the static library is unreachable from any of those dllexport'd symbols, does it matter if the linker doesn't link it in to the DLL? No, because it is unreachable!

Thus, I fail to see how bundling is necessary for linkage to succeed in the Windows world. Maybe things are different on other platforms, sure, but that doesn't mean we should just ignore how Windows works.

A .lib can be a static library or a dynamic library. A .lib can either be bundled into the .rlib or passed to the linker independently. Here we have two boolean situations, so 2*2 = 4 possible combinations. All 4 combinations are legal in the Windows world.

If you take a.lib and b.lib and merge them together first and then pass them to the linker, the end result is the same as if you passed a.lib and b.lib separately to the linker. Due to this, bundling on its own does not actually change how things are linked, and is nothing more than a convenience in that you have fewer things to pass to link.exe.

When a .obj is passed to the linker, its symbols are strong, when a .lib is passed to the linker, its symbols are weak. Strong symbols must be unique or a linker error occurs. Weak symbols are allowed to be defined multiple times. Thus there is one case where bundling does make a difference and that is when you either pass a .obj directly to the linker or first bundle it into a .lib. In the former case if a symbol conflicts with another strong symbol the linker will error, in the latter case it won't error. So I do see the need for an option to bundle an object instead of passing it on to the linker, although this can very easily be replaced by just shoving the .obj into a .lib yourself beforehand instead of expecting rustc to do it for you.

The convenience argument for providing bundling doesn't seem that strong since in a cargo based world, cargo and rustc can keep track of all inputs you supply and make sure they all hang around and get passed on to the linker eventually.

Yes, I understand other platforms behave differently and have their own differing behavior. And yes I understand that Rust is trying to provide things in as cross-platform a manner as possible. However, when it comes to linking, Rust is a native language that integrates into existing native toolchains, so we must respect the large differences in how linking is done across platforms.

19

Actually, that doesn't seem to be the case: http://blogs.msdn.com/b/oldnewthing/archive/2014/03/21/10509670.aspx

The linker ignores dllexport attributes. However, you're correct that this applies equally to both bundled and non-bundled static libraries.

20

But that's the point, why do we need to have two versions of the library? The closest translation to C that I can come up with of my example above works, because it only includes one copy of the library (despite it being listed twice on the link line):

$ cat native.c
#include <stdio.h>
void hello(void) { printf("Hello\n"); }
void world(void) { printf("World\n"); }
$ cat dependency.c 
void hello(void);
void say_hello() { hello(); }
$ cat executable.c 
void say_hello();
void world(void);

int main() {
  say_hello();
  world();
  return 0;
}
$ gcc -fPIC native.c -c ; ar cru libnative.a native.o
$ gcc -fPIC dependency.c -c ; ar cru libdependency.a dependency.o
$ DEPENDENCY_LIBS='-L. -ldependency -lnative' # Probably pkg-config or something in a real case
$ gcc executable.c -o executable -L. -lnative ${DEPENDENCY_LIBS}
$ ./executable 
Hello
World

I started looking into this because we can't link to our (dynamic!) system version of LLVM in Gentoo because no one can come up with a sane patch that will correctly pass kind = "static" if the system libraries are static, but not if they are dynamic. See this comment of yours on PR #28327.

If a library isn't bundled then intermediate artifacts don't contain any parts of it, because that's what it means to not be bundled. (Assuming static linkage here, since that's the case with rlibs and is currently the norm with Rust.)

We only end up with multiple copies of the library because of this bundling behavior. With no bundling the library is only included once, in the final linking step.

This point I fully agree with, and I would not want to remove the ability of rustc to perform the kind = "static" bundling in this case.

21

The best solution is to use an explicit DEF file, since that also gives you a chance to do other things like remove the decorations from the function (so that it can be Get­Proc­Addressed).

Rust currently passes an explicit DEF file anyway, so for Rust anything that we choose to dllexport is in fact exported from the DLL.