[Pre-RFC] `#[link(kind = "dylib")]`

[chrisd]

Summary

Allow #[link(kind = "dylib")] without a name.

Motivation

It is fairly common to write bare extern blocks and fill in the library to be linked elsewhere. E.g.

extern "C" {
    fn do_stuff();
}

// elsewhere in the code...
#[link(name = "mylibrary")]
extern {}

// alternatively, the library may be passed on the command line, through a build script, etc
// or even in another crate.

This works for static libraries because a bare extern block assumes kind = "static". Unfortunately there's no way to declare that the linkage should be dynamic.

For ELF binaries this is usually papered over by the loader trying its hardest to make dynamic libraries appear as much like static libraries as possible. Whereas on Windows this does not happen. Windows does however have "import libraries" that contain shims which similarly papers over the difference. So the static function do_stuff is simply:

do_stuff:
    jmp _imp__do_stuff

However, there is no such equivalent for static items so the following will just fail to link:

extern "C" {
    // where "MY_STATIC" is exported by `mylibrary`
    static MY_STATIC: u8;
}

#[link(name = "mylibrary", kind = "dylib")]
extern {}

Guide-level explanation

By default, functions in an extern block use static linkage. So this:

extern C {
    fn do_stuff();
}

Is equivalent to:

#[link(kind = "static")]
extern C {
    fn do_stuff();
}

You can use dynamic linkage instead by adding #[link(kind = "dylib")]

#[link(kind = "dylib")]
extern "C" {
    fn do_stuff();
}

Reference-level explanation

Currently #[link(kind = "...")] will produce an error saying that name is required. Implementing this RFC would mean carving out an exceptional case to allow kind = "static" | "dylib" if it's the only thing in the #[link] attribute. Otherwise the implementation will be the same except that no library is pushed to the list of native libs.

Drawbacks

Why should we not do this???

Rationale and alternatives

Making overrides more powerful may be an alternative. For example, allowing crates to define where a symbol comes from and how it should be linked independently of defining that symbol.

However, even if that were implemented, I think providing a simple option for the simple case is worthwhile.

Prior art

Windows MSVC (and compatible) have __declspec( dllimport ) which you can use to "import functions, data, and objects [from] a DLL" without adding an import name there.

Unresolved questions

N/A

Future possibilities

I don't expect this will need to be expanded further except perhaps to add more kinds, if it makes sense to do so in the future.

[chrisd]

See also Correctly handle dllimport on Windows · Issue #27438 · rust-lang/rust · GitHub

[Vorpal]

Maybe I'm missing something, but I thought only ELF had a global name space for symbols? My understanding was that both PE and MachO requires you to specify which library you import a given symbol from?

How would that work with this proposal? (I really only know ELF well myself, so I could have misunderstood this completely).

[jrose]

Mach-O records the source library at link time, not compile (or load). I don’t even know if there’s a way to override it in object files. I don’t know about PE.

[chrisd]

For Windows, deciding on which import corresponds to which DLL is the responsibility of an import library (a static library that describes part of the import table). This is the library that's linked in #[link(name = ...)], You never directly link a DLL. Instead you link the symbols provided by the import library (which may or may not be named similarly to the DLL).

Note that on all platforms the current situation is that the name is completely unconnected to the functions/statics in the extern block. You can put the wrong name on the wrong block or put the right name on an empty block and it all works out so long as the linker eventually gets everything it needs.

The one exception is kind = "raw-dylib" on Windows where you need to specify the actual DLL name on the extern block containing the items. This is because it builds the import library itself.

[CAD97]

@chrisd

So the static function do_stuff is simply:

do_stuff:
    jmp _imp__do_stuff

@chrisd

You never directly link a DLL. Instead you link the symbols provided by the import library (which may or may not be named similarly to the DLL).

My understanding was that import libraries are essentially a list of aliases, saying that when the statically linked object names the symbol printf, that actually needs to look in ucrt-api-stdio-v1.dll. I suppose that may be an understanding after linker optimizations, and the actual object file contains a jump thunk; I didn't know those details yet. (The UCRT utilizes PE DLL symbol forwarding in a fun way.)

@chrisd

However, there is no such equivalent for static items so the following will just fail to link:

A simple explanation of how static items get linked would help to explain why #[link(kind = "dylib")] is necessary, but the link name isn't.

It would also probably be good to provide some simple guidance on when each form of #[link] is expected to be used, when it's likely to incidentally work due to abstracting away the staticlib/dylib differences, and when it can just fail to link on major targets. Although since this is largely target specific, just covering tier 1 (ELF, MACH-O, and PE) is enough.

---

#define __dllimport __declspec(dllimport)

I did some research, and I think I understand how PE dllimport works now. The import library is, in effect, a static library with __dllimport void do_stuff();. This creates a symbol void (*__imp__do_stuff)(); in the import address table (the thing the linker/loader does fixups to). A direct call to do_stuff will generate a dynamic call ptr [__imp__do_stuff], and the symbol do_stuff also exists with jmp ptr [__imp__do_stuff] to enable static calls that didn't specify __dllimport.

(do_stuff is the symbolic address of a jump instruction. __imp__do_stuff is the symbolic address of the (populated at load-time) address of the function compilation.)

Global data works somewhat similarly: __dllimport int data; creates the symbol int *__imp__data; in the import address table, and usage of data is lowered as (*__imp__data) instead. But this time there's no possible thunk for the symbol data, so direct static usage without __dllimport will cause a linker error.

(Places are a construct of the compiler. A symbol is an address (potentially relative to some base). data is the symbolic address of the int. __imp__data is the symbolic address of the (populated at load-time) address of the int. PIC complicates things, fun!)

So, interestingly, if you know ahead of time that you will be linking a DLL, using #[link(kind = "dylib")] for functions can be made slightly more efficient. Or at least reduce the amount of LTO work needed to inline all of the DLL import library function thunks.

Additionally, while theoretically it's possible to import the same symbol name from multiple DLLs into one object by giving them a different local name and then mapping them via the import directory table, I don't know if any existing tooling can do that. Potentially you'd be required to fall back onto using raw oridinals instead of dllimport to avoid symbol name conflicts.

This is opposed to an ELF dynamic linker/loader, which more thoroughly abstracts away the difference between linking to staticlib or dylib binaries, giving them an effectively identical linkage interface, only really differing in when symbol fixup happens (build or runtime linkage resolution).

TL;DR: Windows uses local symbols with a single flat namespace like ELF does, but instead of a startup dynamic loader fixing up those symbols, they're just relative within the PE binary, which has a single table of addresses which gets fixed up for any onload DLL dependencies (using DLL + symbol lookup), and the normal static symbols read from that table. Except for global data addresses, which can't expose a DLL import thunk symbol, thus __declspec(dllimport) is mandatory for global DLL data imports. Which Rust spells #[link(kind = "dylib")] extern.

[bjorn3]

@chrisd

This works for static libraries because a bare extern block assumes kind = "static".

No, it assumes that the symbol originates from a rust crate. Without #[link] rustc won't include any symbol inside the extern block in the list of symbols to export when building a dylib, even when the symbol may be called outside of the dylib, whether through a direct expoet of the extern function, or through a cross-crate inlineable function caling it.

[chrisd]

I mean, if you do:

extern "C" {
    fn a_func();
}

Rust leaves it up to the linker to figure out where a_func is defined.

[chrisd]

@CAD97

My understanding was that import libraries are essentially a list of aliases, saying that when the statically linked object names the symbol printf, that actually needs to look in ucrt-api-stdio-v1.dll. I suppose that may be an understanding after linker optimizations, and the actual object file contains a jump thunk; I didn't know those details yet. (The UCRT utilizes PE DLL symbol forwarding in a fun way.)

It's true that nowadays the import library doesn't literally include a jump thunk because a shorter form of import is used that just lists the essentials and the linker builds most of the rest. Still, in that case the thunk is in the library's symbol table so it's "as-if" it exists in the library even if it's not actually created until link time. Whether the import lib uses long or short form, the DLL does not have to exist anywhere (well, until runtime of course).

The historical reason for functions and statics needing to be annotated is that, on x86, an indirect call or load has a longer instruction length than a direct call or load. So it was not possible to fixup the latter. Either some kind of shim is needed or else you need to go back and change the codegen. That's the only real difference between using __declspec(dllimport) or not (well that and the _imp_ naming convention). There are of course architectures where the instructions are the same length so in theory the linker could just fix those up itself in that case.

Additionally, while theoretically it's possible to import the same symbol name from multiple DLLs into one object by giving them a different local name and then mapping them via the import directory table, I don't know if any existing tooling can do that. Potentially you'd be required to fall back onto using raw oridinals instead of dllimport to avoid symbol name conflicts.

You can just use different symbol names. The DLL name and function name to be imported are just strings in a table, only of use to the loader when searching a DLL's export table. Symbol names are an illusion for the sake of static libs and entirely disappear once the linker is done with them (debug info aside).

This also means there is no global static data. Each "module" (exe or dlls) lives in its own world, has its own import table, etc.

[chrisd]

After thinking some more about it I realise I'm basically re-litigating 1717-dllimport - The Rust RFC Book

So I think I'll rewrite this in terms of being an extension to that.

[CAD97]

@chrisd

So I think I'll rewrite this in terms of being an extension

Interesting—by my reading, there's no reason that #[link(kind)] without #[link(name)] should be forbidden by the RFC as written.

Obviously there's more to it than just what that RFC lays out (link(kind) does have nonzero impact on nonwindows targets, e.g. to decoration of name). But I'd probably recommend sticking in a clause that an incorrect kind is considered erroneous, in that it may result in ignoring the kind or a linkage error, but not arbitrary UB.

Especially with the move towards unsafe extern blocks and accepting the reality that incorrect signatures in extern blocks can actually cause UB even when unused.

[mathstuf]

@Vorpal

My understanding was that both PE and MachO requires you to specify which library you import a given symbol from?

Mach-O has the "flat namespace" which is basically "look it up globally". Python interpreters should provide libpython symbols this way so that modules can get the runtime symbols "automagically" rather than actually linking to a (specific) libpython on macOS. FWIW, I've written a patch to say "for any symbol which resolves to library X, instead load it from the flat namespace" instead of having to say -undefined dynamic_lookup which hides warnings about unprovided symbols or a massive list of -U flags.