# Refining cross-platform crt-static semantics

**URL:** <https://internals.rust-lang.org/t/refining-cross-platform-crt-static-semantics/5085>\
**Category:** compiler\
**Created:** [April 12, 2017, 9:57pm UTC](https://internals.rust-lang.org/t/refining-cross-platform-crt-static-semantics/5085 "2017-04-12T21:57:12Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Shizmob](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/shizmob/32/2238_2.png) [@Shizmob](https://internals.rust-lang.org/u/Shizmob)\
**Post date:** [April 12, 2017, 9:57pm UTC](https://internals.rust-lang.org/t/refining-cross-platform-crt-static-semantics/5085/1 "2017-04-12T21:57:13Z")

</div>

_Note: I wasn't sure whether to place this under language design or compiler, since it's a bit of both._

## Background

We've been working on packaging and distributing Rust for the musl-native [Alpine Linux](https://alpinelinux.org) distribution. Our perspective as such is mostly influenced by this. Thanks to the wonderful [crt-static RFC](https://github.com/rust-lang/rfcs/blob/master/text/1721-crt-static.md), Rust now has the possibility to switch between a static and dynamic C runtime at... runtime.

Recently, [a pull request has emerged](https://github.com/rust-lang/rust/pull/40113) to add support for dynamically linking [musl](http://www.musl-libc.org/) target binaries. We based our work on packaging Rust to Alpine partially off this PR, as we as a distribution want to simply distribute dynamically-linked binaries. When working further off these changes to solve some bugs, issues and errors in the compilation and usage of Rust and Cargo, we ran into some issues that challenge the definition of the `crt-static` target feature, especially given its main other usage: Windows MSVC targets.

## CRT in one process

Between Windows and the \*nix systems that I know of, there is one relevant fundamental difference in their approach to C runtimes: \*_in Windows, dynamic libraries can have their own statically linked C runtimes. On nix, this is not possible: there should be a single C runtime within the entire process_.

What does this mean with regards to `crt-static`? It means, for musl targets, that `crt-static` implies that the _the entire final output_ is linked statically: there can be no dynamic libraries with their own private C runtime as a result of the situation above. However, this conflicts with the situation for MSVC, where this is perfectly valid and I'd presume sometimes even appropriate approach. Thus, this is a difference in `crt-static` semantics that needs to be resolved.

I have thought a bit about this, and in my opinion there seem to be two reasonable solutions to this:

- Accept the difference in `crt-static` semantics between platforms (it implying a complete static link on \*nix targets, and not on Windows targets);
- Add a new target feature (maybe simply `static` or `full-static`?), which on \*nix targets would be implied by `crt-static`.

The result of this fully static linkage, whether target-specific or implied by a new target feature, would be that all `unknown`-marked libraries (no specific linkage preference) would be linked statically instead of the default, dynamically, and that (implementation detail) `-static` be added to the linker command line on the GNU linker through a newly added `Linker::static_executable()`. Static linking binaries with musl works as well as per the current situation (of course, since that is currently the main usage for the musl target pre-PR above), but this is more out of luck with the final linker arguments and seems fragile.

Our own patches currently assume the former for simplicitly, but as our aim is to get all relevant patches upstream, resolving this in the way the community feels best has our preference.

## dylibs and `crt-static`

In the situation above, dylibs can obviously not work with `crt-static`. As I understood from @retep998, they do not always work properly on MSVC with `crt-static` either, but are needed nonetheless:

> \<WindowsBunny\> Shiz: One interesting consequence of the static CRT thing on windows is that rust dylibs are horrible. If you ever write any function in a rust dylib which relies on CRT state, you have to make sure it _never_ gets inlined or monomorphized, or it will be using CRT state from the _wrong_ DLL \<WindowsBunny\> Moral of the story, never use rust dylibs \<WindowsBunny\> cdylib is fine though

> \<WindowsBunny\> Shiz: Note that msvc dylibs work fine with crt-static  
> \<WindowsBunny\> Shiz: It's only when you rely on CRT state which Rust normally does not do  
> \<WindowsBunny\> Shiz: using crt-static with dylibs is actually an import thing for rustc, since it is split into a bunch of dylibs and having it not rely on the VC++ redistributable is important

Forbidding/ignoring `dylib` builds on `crt-static` targets where it absolutely will not work, such as on \*nix, may be an interesting avenue to explore, as it may just irritate users with very confusing link-time errors as below, and possibly break Cargo-powered packages that specify a `dylib`:

```rust
   Compiling foo v0.0.1 (file:///home/builder/alpine-aports/testing/cargo/src/cargo-0.17.0/target/cit/t69/foo)
     Running `rustc --crate-name foo src/lib.rs --crate-type dylib --crate-type rlib --emit=dep-info,link -C debuginfo=2 -C metadata=5c5882d5b8d32bd0 --out-dir /home/builder/alpine-aports/testing/cargo/src/cargo-0.17.0/target/cit/t69/foo/target/debug/deps -L dependency=/home/builder/alpine-aports/testing/cargo/src/cargo-0.17.0/target/cit/t69/foo/target/debug/deps`
error: linking with `cc` failed: exit code: 1
  |
  = note: "cc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-Wl,--eh-frame-hdr" "-m64" "-L" "/usr/lib/rustlib/x86_64-unknown-linux-musl/lib" "/home/builder/alpine-aports/testing/cargo/src/cargo-0.17.0/target/cit/t69/foo/target/debug/deps/foo.0.o" "-o" "/home/builder/alpine-aports/testing/cargo/src/cargo-0.17.0/target/cit/t69/foo/target/debug/deps/libfoo.so" "/home/builder/alpine-aports/testing/cargo/src/cargo-0.17.0/target/cit/t69/foo/target/debug/deps/foo.metadata.o" "-nodefaultlibs" "-Wl,-(" "-L" "/home/builder/alpine-aports/testing/cargo/src/cargo-0.17.0/target/cit/t69/foo/target/debug/deps" "-L" "/usr/lib/rustlib/x86_64-unknown-linux-musl/lib" "-Wl,-Bstatic" "-Wl,--whole-archive" "/tmp/rustc.eou9jVOLyku3/libstd-9767eb92afceb88d.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/tmp/rustc.eou9jVOLyku3/librand-3942a5633368c16e.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/tmp/rustc.eou9jVOLyku3/libcollections-ae24cbc916c89b63.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/tmp/rustc.eou9jVOLyku3/libstd_unicode-e03fe48e8667e45a.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/tmp/rustc.eou9jVOLyku3/libpanic_unwind-291a6fecc34131a5.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/tmp/rustc.eou9jVOLyku3/libunwind-8caeecdfbe801a4a.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/tmp/rustc.eou9jVOLyku3/liballoc-ff4a0ee7aa04ad06.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/tmp/rustc.eou9jVOLyku3/liballoc_system-ebd1d2bcdd8dd0ed.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/tmp/rustc.eou9jVOLyku3/liblibc-675e4236817c419c.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/tmp/rustc.eou9jVOLyku3/libcore-ed12715d20c4c8c1.rlib" "-Wl,--no-whole-archive" "/tmp/rustc.eou9jVOLyku3/libcompiler_builtins-c882c2ddbd3077dc.rlib" "-l" "unwind" "-l" "c" "-Wl,-)" "-shared"
  = note: /usr/lib/gcc/x86_64-alpine-linux-musl/6.3.0/../../../../x86_64-alpine-linux-musl/bin/ld: /usr/lib/gcc/x86_64-alpine-linux-musl/6.3.0/../../../../lib/libc.a(sigaction.lo): relocation R_X86_64_PC32 against protected symbol `__restore_rt' can not be used when making a shared object
          /usr/lib/gcc/x86_64-alpine-linux-musl/6.3.0/../../../../x86_64-alpine-linux-musl/bin/ld: final link failed: Bad value
          collect2: error: ld returned 1 exit status

```

## Stabilisation of `crt-static`

One final inquiry I would like to make if it's the intention for the `crt-static` target feature (and a possible second feature as listed above) to be stabilised. Evidently, we feel that being able to flip between being able to compile static and dynamic objects at run-time is an important feature for any compiler, and would like to see it stabilised so we do not have to patch the feature gate out ourselves and possibly create confusion among end-users. 🙂

## Addendum: overview of our patches

This part is mostly an overview of our current patches and what they do to make both dynamic and static linking on a musl-native system work, and is added more for those who are interested in reading it than a direct contribution the discussion. Our patches can be found at [https://github.com/alpinelinux/aports/tree/master/testing/rust](https://github.com/alpinelinux/aports/tree/master/testing/rust), and I'll be describing the functional changes in them file-by-file, for the patches that are relevant to Rust upstream:

- `support-dynamically-linked-musl.patch`: A modified version of [PR #40133](https://github.com/rust-lang/rust/pull/40113), fixing more compilation errors and bugs we ran into:
  - Allow `crt-static` value to be described during bootstrap, so a non-`crt-static` rustc can be compiled using a default `crt-static` bootstrap rustc, and vice-versa;
  - Remove `mips` opt-out of default-static linking for musl, now that it can be toggled easily;
  - Remove imposed dynamic linking restrictions (`dynamic_linking`, `rpath`, `position_independent_executables`) from the musl target;
  - Infer a default musl root of `/usr` for musl-native hosts;
  - (change from the version in the PR): no longer always statically link `libunwind` on musl: instead, do not state a linking preference and let the final linking step decide whether to link statically or dynamically;
  - Add `musl-root` support for MIPS targets;
  - In `libunix`, remove the different `#[link]` statements for deciding how to link `libc` and simply do not state any preference as above with `libunwind`;

- `static-pie.patch`: A restoration patch after the above patch that restores static linking ability, and (relevant to us) introduces static-PIE compatibility:
  - Add `static_position_independent_executables` target feature, indicating support for static PIE binaries, and enable it for `musl` targets;
  - Create fully static binaries in the final executable linking step if the `crt-static` target feature is set;
  - Link libraries with no linkage preference set as static if the `crt-static` target feature is set;
  - Do final link using group linking (using `-Wl,-(` and `-Wl,-)` on the GNU linker, unneeded on the MSVC linker) to prevent dependency issues during a static link (was done solely on the musl target before, but it's a useful general feature);
  - Do not link dylibs when the `crt-static` target feature is set;

- `fix-linux_musl_base.patch`: Makes modifications to the musl base target as a result of the changes in the previous two patches:
  - Do not copy over and manually link the C runtime libraries and startup files for musl: let the compiler (`musl-gcc` on non-musl-native targets) handle this by also removing the passed `-nostdlib` flag, as it's far better at this, and may differ in what it wants to add then what Rust thinks is appropriate, leading to breakage if Rust decides this for itself;
  - Remove the manual `-Wl,-(` and `-Wl,-)` addition to the linker flags in favour of the group linking feature added in `static-pie.patch`;

- `fix-jemalloc-musl.patch`: Fix jemalloc compilation on musl, as an unprefixed jemalloc will conflict with the system allocator in dynamically linked musl binaries (already upstreamed and merged in [PR #41168](https://github.com/rust-lang/rust/pull/41168))
- `allow-crt-static-on-stable.patch`: Un-gate the `crt-static` target feature in our stable release so users can switch between static and dynamic compilation at run-time;
- `change-rpath-to-rustlib.patch` and `force-rpath-on-prefer-dynamic.patch`: Not important, but possibly interesting: our build removes the Rust libraries in `/usr/lib` and only refers to the ones in `/usr/lib/rustlib/$arch/lib` to save space, as they are otherwise identical: this patch makes sure the compiler adds the appropriate `RPATH` directives to this directory when linking dynamically.

## Summary

Or, this post in a few bullet points:

1. With support for dynamically linked musl upcoming, `crt-static` has possibly different semantics between Windows and \*nix platforms with regards to implications if the entire result should be linked statically;
2. As `dylib` support seems (architecturally) broken on \*nix targets with `crt-static`, it may be a good idea to reject or ignore `dylib` compilations when this target feature is enabled;
3. We'd like to propose stabilisation of the `crt-static` feature and a possible extra feature that may arise as a result of bullet point #1, when this issue is resolved;
4. With our patches, we now have a working Rust compiler on a musl-native target OS that can compile both working static and dynamic binaries, with PIE support for static binaries!

_Edited 00:24 CET to correct insights on the necessity of dylibs with `crt-static` on MSVC._

---

<div class="post-metadata">

**Author:** ![binarycrusader](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/binarycrusader/32/2051_2.png) [@binarycrusader](https://internals.rust-lang.org/u/binarycrusader)\
**Post date:** [April 14, 2017, 12:15am UTC](https://internals.rust-lang.org/t/refining-cross-platform-crt-static-semantics/5085/2 "2017-04-14T00:15:54Z")

</div>

I’d just like to mention that in addition to all of this crt-static is not actually possible at all on some platforms. For example, Solaris hasn’t shipped a static libc in many years.

Conversely, my recollection is some Linux variants _only_ ship a static libc!

As such, it’s important that these things are generally advisory rather than compulsory, since otherwise, rust code then becomes unbuildable without modification for those platforms.

---

<div class="post-metadata">

**Author:** ![Shizmob](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/shizmob/32/2238_2.png) [@Shizmob](https://internals.rust-lang.org/u/Shizmob)\
**Post date:** [April 17, 2017, 3:20pm UTC](https://internals.rust-lang.org/t/refining-cross-platform-crt-static-semantics/5085/3 "2017-04-17T15:20:41Z")

</div>

Definitely - one of the points addressed by the [RFC](https://github.com/rust-lang/rfcs/blob/master/text/1721-crt-static.md) is that using a static C runtime was before fixed per platform (e.g. always `true` on musl targets) and could not be changed at run-time at all, adding the `crt-static` target feature to address this. My added suggestions only operate from the basis of `crt-static` being enabled/disabled at runtime, not being fixed per platform.

---

<div class="post-metadata">

**Author:** ![system](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/system/32/14092_2.png) [@system](https://internals.rust-lang.org/u/system)\
**Post date:** [March 25, 2019, 8:28am UTC](https://internals.rust-lang.org/t/refining-cross-platform-crt-static-semantics/5085/4 "2019-03-25T08:28:22Z")

</div>

Closed on March 25, 2019, 8:28am UTC

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.