Bootstrap fails with linker error while running on non-NixOS Nix environment (WSL2 Ubuntu)

Hi. I'm trying to build the Rust compiler using toolchains installed by Nix on non-NixOS Linux (WSL2 Ubuntu). However, when I run ./x.py, the command fails to bootstrap with the following error:

[nix-shell:~/rust]$ ./x.py
info: Downloading and building bootstrap before processing --help
      command. See src/bootstrap/README.md for help with common
      commands.
Updating only changed submodules
Submodules updated in 0.01 seconds
   Compiling proc-macro-error v1.0.4
   Compiling serde v1.0.125
error: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /home/pan/rust/build/bootstrap/debug/deps/libproc_macro_error_attr-d5982b949b70dcd0.so)
   --> /home/pan/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro-error-1.0.4/src/lib.rs:284:9
    |
284 | pub use proc_macro_error_attr::proc_macro_error;
    |         ^^^^^^^^^^^^^^^^^^^^^

error: aborting due to previous error

error: could not compile `proc-macro-error`

To learn more, run the command again with --verbose.
warning: build failed, waiting for other jobs to finish...
error: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /home/pan/rust/build/bootstrap/debug/deps/libserde_derive-913893717e9d98bd.so)
   --> /home/pan/.cargo/registry/src/github.com-1ecc6299db9ec823/serde-1.0.125/src/lib.rs:293:1
    |
293 | extern crate serde_derive;
    | ^^^^^^^^^^^^^^^^^^^^^^^^^^

error: aborting due to previous error

error: build failed
failed to run: /home/pan/rust/build/x86_64-unknown-linux-gnu/stage0/bin/cargo build --manifest-path /home/pan/rust/src/bootstrap/Cargo.toml

I am confused because the error message points to the system libc, but when I use ldd it points to the libc managed by Nix.

[nix-shell:~/rust]$ ldd build/bootstrap/debug/deps/libserde_derive-913893717e9d98bd.so
        linux-vdso.so.1 (0x00007ffc33bfb000)
        libgcc_s.so.1 => /nix/store/v8q6nxyppy1myi3rxni2080bv8s9jxiy-glibc-2.32-40/lib/libgcc_s.so.1 (0x00007f09c0618000)
        libpthread.so.0 => /nix/store/v8q6nxyppy1myi3rxni2080bv8s9jxiy-glibc-2.32-40/lib/libpthread.so.0 (0x00007f09c05f7000)
        libdl.so.2 => /nix/store/v8q6nxyppy1myi3rxni2080bv8s9jxiy-glibc-2.32-40/lib/libdl.so.2 (0x00007f09c05f2000)
        libc.so.6 => /nix/store/v8q6nxyppy1myi3rxni2080bv8s9jxiy-glibc-2.32-40/lib/libc.so.6 (0x00007f09c0431000)
        /nix/store/v8q6nxyppy1myi3rxni2080bv8s9jxiy-glibc-2.32-40/lib64/ld-linux-x86-64.so.2 (0x00007f09c0d5e000)
[nix-shell:~/rust]$ ldd build/x86_64-unknown-linux-gnu/stage0/bin/cargo
        linux-vdso.so.1 (0x00007ffe47739000)
        libgcc_s.so.1 => /nix/store/v8q6nxyppy1myi3rxni2080bv8s9jxiy-glibc-2.32-40/lib/libgcc_s.so.1 (0x00007fa0fda28000)
        librt.so.1 => /nix/store/v8q6nxyppy1myi3rxni2080bv8s9jxiy-glibc-2.32-40/lib/librt.so.1 (0x00007fa0fda1d000)
        libpthread.so.0 => /nix/store/v8q6nxyppy1myi3rxni2080bv8s9jxiy-glibc-2.32-40/lib/libpthread.so.0 (0x00007fa0fd9fc000)
        libm.so.6 => /nix/store/v8q6nxyppy1myi3rxni2080bv8s9jxiy-glibc-2.32-40/lib/libm.so.6 (0x00007fa0fd8b9000)
        libdl.so.2 => /nix/store/v8q6nxyppy1myi3rxni2080bv8s9jxiy-glibc-2.32-40/lib/libdl.so.2 (0x00007fa0fd8b4000)
        libc.so.6 => /nix/store/v8q6nxyppy1myi3rxni2080bv8s9jxiy-glibc-2.32-40/lib/libc.so.6 (0x00007fa0fd6f3000)
        /lib64/ld-linux-x86-64.so.2 => /nix/store/v8q6nxyppy1myi3rxni2080bv8s9jxiy-glibc-2.32-40/lib64/ld-linux-x86-64.so.2 (0x00007fa0feaad000)

Do you have any idea what is the cause of this problem?

Glibc uses symbol versioning. This means that if you link against a certain version of glibc, you must use at least this version at runtime. NixOS likely has a more recent version of glibc than Ubuntu. This means that you can't use executables built on NixOS on Ubuntu. The standard solution for this is building inside a container (or nix environment I think) that has an older glibc version installed. The rustc CI uses CentOS I think as they use a pretty old version of glibc.

1 Like

I guess I'm hitting this:

According to these issues, the cause of this problem seems to be the mismatch between glibc used for building proc macros and used by rustc's dlopen invocation. However, both proc macro and rustc seem to be linked against the same glibc (installed by Nix), and this glibc does support GLIBC_2_32:

[nix-shell:~/rust]$ ldd ./build/x86_64-unknown-linux-gnu/stage0/bin/rustc
        linux-vdso.so.1 (0x00007fd499819000)
        librustc_driver-07fada1ea7dc8755.so => /home/pan/rust/./build/x86_64-unknown-linux-gnu/stage0/bin/../lib/librustc_driver-07fada1ea7dc8755.so (0x00007fd495a7f000)
        libstd-55e6ef8aa4a5856f.so => /home/pan/rust/./build/x86_64-unknown-linux-gnu/stage0/bin/../lib/libstd-55e6ef8aa4a5856f.so (0x00007fd495703000)
        libpthread.so.0 => /nix/store/v8q6nxyppy1myi3rxni2080bv8s9jxiy-glibc-2.32-40/lib/libpthread.so.0 (0x00007fd4956e2000)
        librt.so.1 => /nix/store/v8q6nxyppy1myi3rxni2080bv8s9jxiy-glibc-2.32-40/lib/librt.so.1 (0x00007fd4956d7000)
        libdl.so.2 => /nix/store/v8q6nxyppy1myi3rxni2080bv8s9jxiy-glibc-2.32-40/lib/libdl.so.2 (0x00007fd4956d2000)
        libc.so.6 => /nix/store/v8q6nxyppy1myi3rxni2080bv8s9jxiy-glibc-2.32-40/lib/libc.so.6 (0x00007fd495511000)
        libLLVM-12-rust-1.54.0-beta.so => /home/pan/rust/./build/x86_64-unknown-linux-gnu/stage0/bin/../lib/../lib/libLLVM-12-rust-1.54.0-beta.so (0x00007fd4902e8000)
        libgcc_s.so.1 => /nix/store/v8q6nxyppy1myi3rxni2080bv8s9jxiy-glibc-2.32-40/lib/libgcc_s.so.1 (0x00007fd4902ce000)
        libm.so.6 => /nix/store/v8q6nxyppy1myi3rxni2080bv8s9jxiy-glibc-2.32-40/lib/libm.so.6 (0x00007fd49018b000)
        /lib64/ld-linux-x86-64.so.2 => /nix/store/v8q6nxyppy1myi3rxni2080bv8s9jxiy-glibc-2.32-40/lib64/ld-linux-x86-64.so.2 (0x00007fd49981b000)
        libz.so.1 => not found
[nix-shell:~/rust]$ ldd ./build/bootstrap/debug/deps/libproc_macro_error_attr-d5982b949b70dcd0.so
        linux-vdso.so.1 (0x00007ffc6fffc000)
        libgcc_s.so.1 => /nix/store/v8q6nxyppy1myi3rxni2080bv8s9jxiy-glibc-2.32-40/lib/libgcc_s.so.1 (0x00007f1538de8000)
        libpthread.so.0 => /nix/store/v8q6nxyppy1myi3rxni2080bv8s9jxiy-glibc-2.32-40/lib/libpthread.so.0 (0x00007f1538dc7000)
        libdl.so.2 => /nix/store/v8q6nxyppy1myi3rxni2080bv8s9jxiy-glibc-2.32-40/lib/libdl.so.2 (0x00007f1538dc2000)
        libc.so.6 => /nix/store/v8q6nxyppy1myi3rxni2080bv8s9jxiy-glibc-2.32-40/lib/libc.so.6 (0x00007f1538c01000)
        /nix/store/v8q6nxyppy1myi3rxni2080bv8s9jxiy-glibc-2.32-40/lib64/ld-linux-x86-64.so.2 (0x00007f1539055000)

I don't get why system libc (/lib/x86_64-linux-gnu/libc.so.6) is used and appears in the error message in the first place😅

I don't think I'm using binaries built on NixOS but bootstrapping binaries automatically downloaded by x.py.

Yes, at runtime. However, your error:

indicates that the system libc is being used during the build. I suspect that your Nix environment is either not set up properly or rpath entries are missing.

1 Like

I found that LD_TRACE_LOADED_OBJECTS=1 and ldd return different libc paths for the bootstrap rustc😂

[nix-shell:~/rust]$ LD_TRACE_LOADED_OBJECTS=1 ./build/x86_64-unknown-linux-gnu/stage0/bin/rustc
        linux-vdso.so.1 (0x00007ffd63d88000)
        librustc_driver-07fada1ea7dc8755.so => /home/pan/rust/build/x86_64-unknown-linux-gnu/stage0/bin/../lib/librustc_driver-07fada1ea7dc8755.so (0x00007f5b8e049000)
        libstd-55e6ef8aa4a5856f.so => /home/pan/rust/build/x86_64-unknown-linux-gnu/stage0/bin/../lib/libstd-55e6ef8aa4a5856f.so (0x00007f5b8dccd000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f5b8dca2000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f5b8dc97000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f5b8dc91000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f5b8da9f000)
        libLLVM-12-rust-1.54.0-beta.so => /home/pan/rust/build/x86_64-unknown-linux-gnu/stage0/bin/../lib/../lib/libLLVM-12-rust-1.54.0-beta.so (0x00007f5b88876000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f5b8885b000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f5b8870c000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f5b9196d000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f5b886f0000)

[nix-shell:~/rust]$ ldd ./build/x86_64-unknown-linux-gnu/stage0/bin/rustc
        linux-vdso.so.1 (0x00007ffca0eb4000)
        librustc_driver-07fada1ea7dc8755.so => /home/pan/rust/./build/x86_64-unknown-linux-gnu/stage0/bin/../lib/librustc_driver-07fada1ea7dc8755.so (0x00007f5119adc000)
        libstd-55e6ef8aa4a5856f.so => /home/pan/rust/./build/x86_64-unknown-linux-gnu/stage0/bin/../lib/libstd-55e6ef8aa4a5856f.so (0x00007f5119760000)
        libpthread.so.0 => /nix/store/v8q6nxyppy1myi3rxni2080bv8s9jxiy-glibc-2.32-40/lib/libpthread.so.0 (0x00007f511973f000)
        librt.so.1 => /nix/store/v8q6nxyppy1myi3rxni2080bv8s9jxiy-glibc-2.32-40/lib/librt.so.1 (0x00007f5119734000)
        libdl.so.2 => /nix/store/v8q6nxyppy1myi3rxni2080bv8s9jxiy-glibc-2.32-40/lib/libdl.so.2 (0x00007f511972f000)
        libc.so.6 => /nix/store/v8q6nxyppy1myi3rxni2080bv8s9jxiy-glibc-2.32-40/lib/libc.so.6 (0x00007f511956e000)
        libLLVM-12-rust-1.54.0-beta.so => /home/pan/rust/./build/x86_64-unknown-linux-gnu/stage0/bin/../lib/../lib/libLLVM-12-rust-1.54.0-beta.so (0x00007f5114345000)
        libgcc_s.so.1 => /nix/store/v8q6nxyppy1myi3rxni2080bv8s9jxiy-glibc-2.32-40/lib/libgcc_s.so.1 (0x00007f511432b000)
        libm.so.6 => /nix/store/v8q6nxyppy1myi3rxni2080bv8s9jxiy-glibc-2.32-40/lib/libm.so.6 (0x00007f51141e8000)
        /lib64/ld-linux-x86-64.so.2 => /nix/store/v8q6nxyppy1myi3rxni2080bv8s9jxiy-glibc-2.32-40/lib64/ld-linux-x86-64.so.2 (0x00007f511d873000)
        libz.so.1 => not found

So yes, system libc is used by rustc. Then I need to find a way to override these paths...

Bootstrap patches rustc automatically on NixOS. It probably doesn't do this for you. The normal way to patch it for Nix would be using patchelf to override the interpreter and append the rpath. It is probably easier to convince bootstrap that you are on NixOS though. I think removing

will do the trick.

1 Like

Commenting out these lines did the trick. Thanks! It makes sense that the bootstrap binaries were not patched because my environment is a usual Linux with Nix, not NixOS.

Now I am wondering if it would be a good idea to send a patch to add a config that overrides this check to compile rustc with Nix's toolchain.