Refining cross-platform crt-static semantics

[Shizmob]

Note: I wasn't sure whether to place this under language design or compiler, since it's a bit of both.

Background

We've been working on packaging and distributing Rust for the musl-native Alpine Linux distribution. Our perspective as such is mostly influenced by this. Thanks to the wonderful crt-static RFC, Rust now has the possibility to switch between a static and dynamic C runtime at... runtime.

Recently, a pull request has emerged to add support for dynamically linking musl target binaries. We based our work on packaging Rust to Alpine partially off this PR, as we as a distribution want to simply distribute dynamically-linked binaries. When working further off these changes to solve some bugs, issues and errors in the compilation and usage of Rust and Cargo, we ran into some issues that challenge the definition of the crt-static target feature, especially given its main other usage: Windows MSVC targets.

CRT in one process

Between Windows and the *nix systems that I know of, there is one relevant fundamental difference in their approach to C runtimes: *in Windows, dynamic libraries can have their own statically linked C runtimes. On nix, this is not possible: there should be a single C runtime within the entire process.

What does this mean with regards to crt-static? It means, for musl targets, that crt-static implies that the the entire final output is linked statically: there can be no dynamic libraries with their own private C runtime as a result of the situation above. However, this conflicts with the situation for MSVC, where this is perfectly valid and I'd presume sometimes even appropriate approach. Thus, this is a difference in crt-static semantics that needs to be resolved.

I have thought a bit about this, and in my opinion there seem to be two reasonable solutions to this:

• Accept the difference in crt-static semantics between platforms (it implying a complete static link on *nix targets, and not on Windows targets);
• Add a new target feature (maybe simply static or full-static?), which on *nix targets would be implied by crt-static.

The result of this fully static linkage, whether target-specific or implied by a new target feature, would be that all unknown-marked libraries (no specific linkage preference) would be linked statically instead of the default, dynamically, and that (implementation detail) -static be added to the linker command line on the GNU linker through a newly added Linker::static_executable(). Static linking binaries with musl works as well as per the current situation (of course, since that is currently the main usage for the musl target pre-PR above), but this is more out of luck with the final linker arguments and seems fragile.

Our own patches currently assume the former for simplicitly, but as our aim is to get all relevant patches upstream, resolving this in the way the community feels best has our preference.

dylibs and crt-static

In the situation above, dylibs can obviously not work with crt-static. As I understood from @retep998, they do not always work properly on MSVC with crt-static either, but are needed nonetheless:

<WindowsBunny> Shiz: One interesting consequence of the static CRT thing on windows is that rust dylibs are horrible. If you ever write any function in a rust dylib which relies on CRT state, you have to make sure it never gets inlined or monomorphized, or it will be using CRT state from the wrong DLL <WindowsBunny> Moral of the story, never use rust dylibs <WindowsBunny> cdylib is fine though

<WindowsBunny> Shiz: Note that msvc dylibs work fine with crt-static
<WindowsBunny> Shiz: It's only when you rely on CRT state which Rust normally does not do
<WindowsBunny> Shiz: using crt-static with dylibs is actually an import thing for rustc, since it is split into a bunch of dylibs and having it not rely on the VC++ redistributable is important

Forbidding/ignoring dylib builds on crt-static targets where it absolutely will not work, such as on *nix, may be an interesting avenue to explore, as it may just irritate users with very confusing link-time errors as below, and possibly break Cargo-powered packages that specify a dylib:

   Compiling foo v0.0.1 (file:///home/builder/alpine-aports/testing/cargo/src/cargo-0.17.0/target/cit/t69/foo)
     Running `rustc --crate-name foo src/lib.rs --crate-type dylib --crate-type rlib --emit=dep-info,link -C debuginfo=2 -C metadata=5c5882d5b8d32bd0 --out-dir /home/builder/alpine-aports/testing/cargo/src/cargo-0.17.0/target/cit/t69/foo/target/debug/deps -L dependency=/home/builder/alpine-aports/testing/cargo/src/cargo-0.17.0/target/cit/t69/foo/target/debug/deps`
error: linking with `cc` failed: exit code: 1
  |
  = note: "cc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-Wl,--eh-frame-hdr" "-m64" "-L" "/usr/lib/rustlib/x86_64-unknown-linux-musl/lib" "/home/builder/alpine-aports/testing/cargo/src/cargo-0.17.0/target/cit/t69/foo/target/debug/deps/foo.0.o" "-o" "/home/builder/alpine-aports/testing/cargo/src/cargo-0.17.0/target/cit/t69/foo/target/debug/deps/libfoo.so" "/home/builder/alpine-aports/testing/cargo/src/cargo-0.17.0/target/cit/t69/foo/target/debug/deps/foo.metadata.o" "-nodefaultlibs" "-Wl,-(" "-L" "/home/builder/alpine-aports/testing/cargo/src/cargo-0.17.0/target/cit/t69/foo/target/debug/deps" "-L" "/usr/lib/rustlib/x86_64-unknown-linux-musl/lib" "-Wl,-Bstatic" "-Wl,--whole-archive" "/tmp/rustc.eou9jVOLyku3/libstd-9767eb92afceb88d.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/tmp/rustc.eou9jVOLyku3/librand-3942a5633368c16e.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/tmp/rustc.eou9jVOLyku3/libcollections-ae24cbc916c89b63.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/tmp/rustc.eou9jVOLyku3/libstd_unicode-e03fe48e8667e45a.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/tmp/rustc.eou9jVOLyku3/libpanic_unwind-291a6fecc34131a5.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/tmp/rustc.eou9jVOLyku3/libunwind-8caeecdfbe801a4a.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/tmp/rustc.eou9jVOLyku3/liballoc-ff4a0ee7aa04ad06.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/tmp/rustc.eou9jVOLyku3/liballoc_system-ebd1d2bcdd8dd0ed.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/tmp/rustc.eou9jVOLyku3/liblibc-675e4236817c419c.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/tmp/rustc.eou9jVOLyku3/libcore-ed12715d20c4c8c1.rlib" "-Wl,--no-whole-archive" "/tmp/rustc.eou9jVOLyku3/libcompiler_builtins-c882c2ddbd3077dc.rlib" "-l" "unwind" "-l" "c" "-Wl,-)" "-shared"
  = note: /usr/lib/gcc/x86_64-alpine-linux-musl/6.3.0/../../../../x86_64-alpine-linux-musl/bin/ld: /usr/lib/gcc/x86_64-alpine-linux-musl/6.3.0/../../../../lib/libc.a(sigaction.lo): relocation R_X86_64_PC32 against protected symbol `__restore_rt' can not be used when making a shared object
          /usr/lib/gcc/x86_64-alpine-linux-musl/6.3.0/../../../../x86_64-alpine-linux-musl/bin/ld: final link failed: Bad value
          collect2: error: ld returned 1 exit status

Stabilisation of crt-static

One final inquiry I would like to make if it's the intention for the crt-static target feature (and a possible second feature as listed above) to be stabilised. Evidently, we feel that being able to flip between being able to compile static and dynamic objects at run-time is an important feature for any compiler, and would like to see it stabilised so we do not have to patch the feature gate out ourselves and possibly create confusion among end-users.

Addendum: overview of our patches

This part is mostly an overview of our current patches and what they do to make both dynamic and static linking on a musl-native system work, and is added more for those who are interested in reading it than a direct contribution the discussion. Our patches can be found at https://github.com/alpinelinux/aports/tree/master/testing/rust, and I'll be describing the functional changes in them file-by-file, for the patches that are relevant to Rust upstream:

• support-dynamically-linked-musl.patch: A modified version of PR #40133, fixing more compilation errors and bugs we ran into:
  • Allow crt-static value to be described during bootstrap, so a non-crt-static rustc can be compiled using a default crt-static bootstrap rustc, and vice-versa;
  • Remove mips opt-out of default-static linking for musl, now that it can be toggled easily;
  • Remove imposed dynamic linking restrictions (dynamic_linking, rpath, position_independent_executables) from the musl target;
  • Infer a default musl root of /usr for musl-native hosts;
  • (change from the version in the PR): no longer always statically link libunwind on musl: instead, do not state a linking preference and let the final linking step decide whether to link statically or dynamically;
  • Add musl-root support for MIPS targets;
  • In libunix, remove the different #[link] statements for deciding how to link libc and simply do not state any preference as above with libunwind;
• static-pie.patch: A restoration patch after the above patch that restores static linking ability, and (relevant to us) introduces static-PIE compatibility:
  • Add static_position_independent_executables target feature, indicating support for static PIE binaries, and enable it for musl targets;
  • Create fully static binaries in the final executable linking step if the crt-static target feature is set;
  • Link libraries with no linkage preference set as static if the crt-static target feature is set;
  • Do final link using group linking (using -Wl,-( and -Wl,-) on the GNU linker, unneeded on the MSVC linker) to prevent dependency issues during a static link (was done solely on the musl target before, but it's a useful general feature);
  • Do not link dylibs when the crt-static target feature is set;
• fix-linux_musl_base.patch: Makes modifications to the musl base target as a result of the changes in the previous two patches:
  • Do not copy over and manually link the C runtime libraries and startup files for musl: let the compiler (musl-gcc on non-musl-native targets) handle this by also removing the passed -nostdlib flag, as it's far better at this, and may differ in what it wants to add then what Rust thinks is appropriate, leading to breakage if Rust decides this for itself;
  • Remove the manual -Wl,-( and -Wl,-) addition to the linker flags in favour of the group linking feature added in static-pie.patch;
• fix-jemalloc-musl.patch: Fix jemalloc compilation on musl, as an unprefixed jemalloc will conflict with the system allocator in dynamically linked musl binaries (already upstreamed and merged in PR #41168)
• allow-crt-static-on-stable.patch: Un-gate the crt-static target feature in our stable release so users can switch between static and dynamic compilation at run-time;
• change-rpath-to-rustlib.patch and force-rpath-on-prefer-dynamic.patch: Not important, but possibly interesting: our build removes the Rust libraries in /usr/lib and only refers to the ones in /usr/lib/rustlib/$arch/lib to save space, as they are otherwise identical: this patch makes sure the compiler adds the appropriate RPATH directives to this directory when linking dynamically.

Summary

Or, this post in a few bullet points:

1. With support for dynamically linked musl upcoming, crt-static has possibly different semantics between Windows and *nix platforms with regards to implications if the entire result should be linked statically;
2. As dylib support seems (architecturally) broken on *nix targets with crt-static, it may be a good idea to reject or ignore dylib compilations when this target feature is enabled;
3. We'd like to propose stabilisation of the crt-static feature and a possible extra feature that may arise as a result of bullet point #1, when this issue is resolved;
4. With our patches, we now have a working Rust compiler on a musl-native target OS that can compile both working static and dynamic binaries, with PIE support for static binaries!

Edited 00:24 CET to correct insights on the necessity of dylibs with crt-static on MSVC.

[binarycrusader]

I’d just like to mention that in addition to all of this crt-static is not actually possible at all on some platforms. For example, Solaris hasn’t shipped a static libc in many years.

Conversely, my recollection is some Linux variants only ship a static libc!

As such, it’s important that these things are generally advisory rather than compulsory, since otherwise, rust code then becomes unbuildable without modification for those platforms.

[Shizmob]

Definitely - one of the points addressed by the RFC is that using a static C runtime was before fixed per platform (e.g. always true on musl targets) and could not be changed at run-time at all, adding the crt-static target feature to address this. My added suggestions only operate from the basis of crt-static being enabled/disabled at runtime, not being fixed per platform.