# Librustc\_driver.so not reproducible **URL:** \ **Category:** compiler\ **Created:** [October 2, 2023, 4:15pm UTC](https://internals.rust-lang.org/t/librustc-driver-so-not-reproducible/19639 "2023-10-02T16:15:46Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![sundeep-kokkonda](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/sundeep-kokkonda/32/10110_2.png) [@sundeep-kokkonda](https://internals.rust-lang.org/u/sundeep-kokkonda)\ **Post date:** [October 2, 2023, 4:15pm UTC](https://internals.rust-lang.org/t/librustc-driver-so-not-reproducible/19639/1 "2023-10-02T16:15:46Z")
Hello, While building rust compiler (in a linux system where host and target are same) between two different build directories, we've seen a reproducibility issue with _librustc\_driver.so_ file and this we are _able to fix_ by using the _--remap-path-prefix_ option usage. But, in our Yocto project we are building the rust compiler and there also we faced the same reproducibility issue (here the host and target are different i.e., the rust compiler will get cross-compiled for a different target). The solution of using _--remap-path-prefix_ is not working here. There a section called '_.rustc_' in _librustc\_driver.so_ is differed and that making the binary comparison failed b/w two build directories. Can I get some info from community on below... 1. What is this .rustc section and what it contains? (I couldn't disasemble this section and coudn't find much info on doc as well) 2. Why this section is dependent in build path? 3. What could be the reasons for --remap-path-prefix not working? (I can see this option is passed during Yocto build as well for rust compilation) Thanks... ---
**Author:** ![cuviper](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/cuviper/32/1897_2.png) [@cuviper](https://internals.rust-lang.org/u/cuviper)\ **Post date:** [October 2, 2023, 5:54pm UTC](https://internals.rust-lang.org/t/librustc-driver-so-not-reproducible/19639/2 "2023-10-02T17:54:02Z")
> [@sundeep-kokkonda](#): > > (here the host and target are different i.e., the rust compiler will get cross-compiled for a different target) Are the respective host and target at least consistent between the builds you're comparing? I know that Cargo includes the host in its crate metadata hash, so you will end up with different library and symbol hashes if you build the same target from different hosts. ---
**Author:** ![Vorpal](https://avatars.discourse-cdn.com/v4/letter/v/aca169/32.png) [@Vorpal](https://internals.rust-lang.org/u/Vorpal)\ **Post date:** [October 2, 2023, 6:52pm UTC](https://internals.rust-lang.org/t/librustc-driver-so-not-reproducible/19639/3 "2023-10-02T18:52:42Z")
> [@cuviper](#): > > I know that Cargo includes the host in its crate metadata hash, so you will end up with different library and symbol hashes if you build the same target from different hosts. This sounds like a terrible idea for having reproducible builds. That should really be opt-in, but is there at least a flag to turn it off? ---
**Author:** ![cuviper](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/cuviper/32/1897_2.png) [@cuviper](https://internals.rust-lang.org/u/cuviper)\ **Post date:** [October 2, 2023, 6:57pm UTC](https://internals.rust-lang.org/t/librustc-driver-so-not-reproducible/19639/4 "2023-10-02T18:57:16Z")
For more context, I have approached the host metadata issue before: > > > Cross-compiling the same target from different hosts should still > produce the s…ame output from rustc, but cargo effects a difference by > hashing the full \`rustc.verbose\_version\`, including \`host: \\`. We > can filter that particular line to allow different hosts to produce the > same target metadata after all. In my case, I ended up working around this in my rpm builds by only building the cross-target parts from x86\_64, but the resulting target libraries are still usable from other hosts. ---
**Author:** ![the8472](https://avatars.discourse-cdn.com/v4/letter/t/0ea827/32.png) [@the8472](https://internals.rust-lang.org/u/the8472)\ **Post date:** [October 2, 2023, 7:19pm UTC](https://internals.rust-lang.org/t/librustc-driver-so-not-reproducible/19639/5 "2023-10-02T19:19:21Z")
In the general case two different rustc builds are not ABI-compatible with each other, e.g. they could be built with different target flags and thus have different call ABIs or with different struct layout randomization. Two different builds producing identical output is the exception, not the norm. ---
**Author:** ![sundeep-kokkonda](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/sundeep-kokkonda/32/10110_2.png) [@sundeep-kokkonda](https://internals.rust-lang.org/u/sundeep-kokkonda)\ **Post date:** [October 3, 2023, 3:59am UTC](https://internals.rust-lang.org/t/librustc-driver-so-not-reproducible/19639/6 "2023-10-03T03:59:55Z")
> [@cuviper](#): > > Are the respective host and target at least consistent between the builds you're comparing? Yes, those are consistent (the host and target are in Yocto are for x86\_64 arch i.e., "x86\_64-poky-linux"). Is this the consistency you asked? And, I am not building the target compiler with different hosts, but in the different build paths in build machine with same host. Also, the rlib's & .so are generated with identical hash b/w two build directories. I can only see the difference in ".rustc" section of the generated objects in different rust build stages. Below is an instance of the diff: (buildX & buildY are the build directories) > _stage0-sysroot: libcstr-94890041bc3cec88.so_ > > buildX: 0000000000000000 g O **.rustc 0000000000000822** rust\_metadata\_cstr\_39367acaf44167b3 > > buildY: 0000000000000000 g O **.rustc 0000000000000820** rust\_metadata\_cstr\_39367acaf44167b3 > stage2: librustc\_driver-62e98be45f5bb4e6.so > > buildX: 0000000000000000 g O **.rustc 00000000000028e1** rust\_metadata\_rustc\_driver\_d1b873a1521f7a0e > > buildY: 0000000000000000 g O **.rustc 00000000000028e6** rust\_metadata\_rustc\_driver\_d1b873a1521f7a0e ---
**Author:** ![cuviper](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/cuviper/32/1897_2.png) [@cuviper](https://internals.rust-lang.org/u/cuviper)\ **Post date:** [October 3, 2023, 2:19pm UTC](https://internals.rust-lang.org/t/librustc-driver-so-not-reproducible/19639/7 "2023-10-03T14:19:03Z")
Ok, then it doesn't look like you're facing the issue I knew, but I don't have a guess for what's causing your difference. ---
**Author:** ![zackw](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/zackw/32/2071_2.png) [@zackw](https://internals.rust-lang.org/u/zackw)\ **Post date:** [October 3, 2023, 3:32pm UTC](https://internals.rust-lang.org/t/librustc-driver-so-not-reproducible/19639/8 "2023-10-03T15:32:27Z")
Can we see a comparison of the output of `readelf -x .rustc [library]` for both versions of librustc\_driver, please? ---
**Author:** ![sundeep-kokkonda](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/sundeep-kokkonda/32/10110_2.png) [@sundeep-kokkonda](https://internals.rust-lang.org/u/sundeep-kokkonda)\ **Post date:** [October 3, 2023, 3:54pm UTC](https://internals.rust-lang.org/t/librustc-driver-so-not-reproducible/19639/9 "2023-10-03T15:54:13Z")
I cannot attach the text of the dump. So, attached the diff image. ![readelf](https://us1.discourse-cdn.com/flex002/uploads/rustlang/original/2X/9/9d5f46aeb235b390e3f989bdbfc4211a9b5c5b25.jpeg) ---
**Author:** ![bjorn3](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/bjorn3/32/2736_2.png) [@bjorn3](https://internals.rust-lang.org/u/bjorn3)\ **Post date:** [October 3, 2023, 7:19pm UTC](https://internals.rust-lang.org/t/librustc-driver-so-not-reproducible/19639/10 "2023-10-03T19:19:51Z")
Could you remove the first 16 bytes (a header added by rustc) and then decompress it? It is [snappy](https://github.com/google/snappy) compressed. That should make it easier to see what the actual change is without getting a cascade of differences. It would also show if it is maybe non-determinism in the compression rather than the data being compressed. By the way starting from rustc 1.73 the crate metadata in dylibs is no longer compressed to avoid a perf regression in a [bugfix](https://github.com/rust-lang/rust/pull/113695). ---
**Author:** ![sundeep-kokkonda](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/sundeep-kokkonda/32/10110_2.png) [@sundeep-kokkonda](https://internals.rust-lang.org/u/sundeep-kokkonda)\ **Post date:** [October 6, 2023, 7:12am UTC](https://internals.rust-lang.org/t/librustc-driver-so-not-reproducible/19639/11 "2023-10-06T07:12:06Z")
@bjorn3 I tried a build with rust 1.73 and still see some compressed data (may be not with Snappy as I couldn't see that string now). How can I decompress this section to a plain text. ![rust173](https://us1.discourse-cdn.com/flex002/uploads/rustlang/original/2X/f/fead35d33d176aba0add7edd57879f96d9a53ed5.jpeg) I tried the previous data (generated with 1.70.0) to decompress with snappy but it was giving some errors like '_unknown start of token: \_' & '_unterminated character literal_'... etc, but that was the data generated by the rust compiler in .so file. ---
**Author:** ![bjorn3](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/bjorn3/32/2736_2.png) [@bjorn3](https://internals.rust-lang.org/u/bjorn3)\ **Post date:** [October 6, 2023, 7:35am UTC](https://internals.rust-lang.org/t/librustc-driver-so-not-reproducible/19639/12 "2023-10-06T07:35:22Z")
There shouldn't be any compression. This place looks like it is the list of dependencies. Could you take `libdigest-.rmeta` for both compilations and show the diff? This is a crate which according to this diff you posted has a different crate hash and it shouldn't have any dependencies beyond the standard library. And could you see if `libcore-.rlib` has any differences already? ---
**Author:** ![sundeep-kokkonda](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/sundeep-kokkonda/32/10110_2.png) [@sundeep-kokkonda](https://internals.rust-lang.org/u/sundeep-kokkonda)\ **Post date:** [October 6, 2023, 9:07am UTC](https://internals.rust-lang.org/t/librustc-driver-so-not-reproducible/19639/13 "2023-10-06T09:07:57Z")
@bjorn3 I compared the `libdigest-.rmeta, libdigest-.rlib & libcore-.rlib` files geneared during stage0&1. Only the `stage0-rustc` artifacts are differed in `.rmeta` file. Since `rmeta` is not a binary I couldn't dump it. Below is the diff. ![rmeta-diff](https://us1.discourse-cdn.com/flex002/uploads/rustlang/original/2X/d/d8b2150812df81a2b68b0c27e09c8f0f3904ab29.jpeg) As per the rust doc, the `rmeta` files do not support linking, since they do not contain compiled object files. And also the doc says, A `dylib` is a platform-specific shared library. It includes the `rustc` [metadata] in a special link section called `.rustc` in a compressed format. Does the `rmeta` changes influence the final binaries, and **why this rmeta section is changing by build path**? Is there any tool available to dump the rmeta files? ---
**Author:** ![bjorn3](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/bjorn3/32/2736_2.png) [@bjorn3](https://internals.rust-lang.org/u/bjorn3)\ **Post date:** [October 6, 2023, 10:50am UTC](https://internals.rust-lang.org/t/librustc-driver-so-not-reproducible/19639/14 "2023-10-06T10:50:18Z")
The `.rmeta` files are identical to the `.rmeta` section in dylibs and the `lib.rmeta` files in rlibs. ---
**Author:** ![bjorn3](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/bjorn3/32/2736_2.png) [@bjorn3](https://internals.rust-lang.org/u/bjorn3)\ **Post date:** [October 6, 2023, 10:52am UTC](https://internals.rust-lang.org/t/librustc-driver-so-not-reproducible/19639/15 "2023-10-06T10:52:11Z")
> [@sundeep-kokkonda](#): > > Is there any tool available to dump the rmeta files? `rustc -Zls /path/to/file` (`rustc -Zls=full /path/to/file` for newer nightly, I extended it in [https://github.com/rust-lang/rust/pull/115735](https://github.com/rust-lang/rust/pull/115735)) can dump some information from the crate metadata of a `.rmeta`, `.rlib` or rust `.so`. Make sure to use the same rustc as the one that originally compiled it. Most information in the crate metadata can't be dumped using `rustc -Zls` though and there is no external tool which works with current rustc versions. ---
**Author:** ![sundeep-kokkonda](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/sundeep-kokkonda/32/10110_2.png) [@sundeep-kokkonda](https://internals.rust-lang.org/u/sundeep-kokkonda)\ **Post date:** [October 9, 2023, 11:52am UTC](https://internals.rust-lang.org/t/librustc-driver-so-not-reproducible/19639/16 "2023-10-09T11:52:49Z")
@bjorn3 , thanks for the info, this update helps me analyze the issue further. I've checked the diff of the metadata info pulled by the `-Zls=y` option for _libdigest_ rmeta file and this differes are in `libcrypto_common-.rmeta, libgeneric_array-.rmeta, libtypenum-.rmeta & libblock_buffer-.rmeta` files. I tried comparing a few other rmeta files also, those are also differed in the same set of files. Out of these, The `libtypenum-.rmeta` has not shown any diff in it's dependencies but still shows a different fingerprint/hash. ![rmeta-Zls](https://us1.discourse-cdn.com/flex002/uploads/rustlang/original/2X/1/1925756d4a9ee424bc631e4f2a4dc6e33163f74c.jpeg) How to understand this fingerprint/hash diff i.e., is it expected to change when a build path of rust sources changed? I had a look into `cargo/src/cargo/core/compiler/fingerprint/mod.rs` file, the comments shows that the fingerprint is depending on the Path of the source file. **Let me explain a bit more detail about my observations.** I built the rust in - `/home/workspace/rustc/buildX & buildY` directories (buildX & buildY are my build directories) which differ in the `librustc_driver-.so` file. (File path - \<\>/x86\_64-unknown-linux-gnu/stage2/lib/) I added a `--remap-path-prefix` command (by providing the build directory paths to replace with a some string) in the `src/bootstrap/builder.rs` in `fn cargo` function and this generated a identical `librustc_driver-.so` file in buildX & buildY directories. The same rust sources I rebuild in another directory - `/home/workspace/extended-path/rustc/buildX & buildY` with the '--remap-path-prefix' command. Now, when I compared the `librustc_driver-.so` in, `/home/workspace/rustc/buildX & /home/workspace/extended-path/rustc/buildX` these are differed again in .rustc section. (And, I read the fingerprint of some libs in two directories and those are entirely different) **So, for me it seems the rust is generating the fingerprint based on build path and which is getting changed when the build path is changed.** **Is there any command which fixes this issue with fingerprint/.rustc sections or it's a bug in rust?** Please let me know if any additional info is needed or anything is unclear. ---
**Author:** ![bjorn3](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/bjorn3/32/2736_2.png) [@bjorn3](https://internals.rust-lang.org/u/bjorn3)\ **Post date:** [October 9, 2023, 12:28pm UTC](https://internals.rust-lang.org/t/librustc-driver-so-not-reproducible/19639/17 "2023-10-09T12:28:37Z")
With `remap-debuginfo` enabled in config.toml everything should be reproducible when changing the source dir location. If that isn't the case like here, that is a bug. By the way I just now noticed that you were manually passing `--remap-path-prefix` instead of enabling `remap-debuginfo`. > [@sundeep-kokkonda](#): > > Out of these, The `libtypenum-.rmeta` has not shown any diff in it's dependencies but still shows a different fingerprint/hash. Could you show the diff of this file? ---
**Author:** ![sundeep-kokkonda](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/sundeep-kokkonda/32/10110_2.png) [@sundeep-kokkonda](https://internals.rust-lang.org/u/sundeep-kokkonda)\ **Post date:** [October 9, 2023, 4:51pm UTC](https://internals.rust-lang.org/t/librustc-driver-so-not-reproducible/19639/18 "2023-10-09T16:51:31Z")
@bjorn3 Thanks a lot for letting me know about `remap-debuginfo` that actually fixed the issue now the `.rustc` section is identical. Here is the `libtypenum-.rmeta` diff. ![libtypenum](https://us1.discourse-cdn.com/flex002/uploads/rustlang/original/2X/1/1c1d2c34f282d9cd82758cb7a7da0368946af326.jpeg) But, I've another section `.dynstr` is still differing, which is holding the absolute `rpath` (This is in our Yocto project, there we are passing the `-Clink-args=-Wl,-rpath,$ORIGIN/../lib`, and this `rpath` is getting adding to the `.dynstr` section) . These `remap-debuginfo` & `--remap-path-prefix` options won't work on entire object file? ---
**Author:** ![bjorn3](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/bjorn3/32/2736_2.png) [@bjorn3](https://internals.rust-lang.org/u/bjorn3)\ **Post date:** [October 9, 2023, 6:04pm UTC](https://internals.rust-lang.org/t/librustc-driver-so-not-reproducible/19639/19 "2023-10-09T18:04:45Z")
> [@sundeep-kokkonda](#): > > Thanks a lot for letting me know about `remap-debuginfo` that actually fixed the issue now the `.rustc` section is identical. Great! I'm honestly surprised it is not the default for the dist profile. > [@sundeep-kokkonda](#): > > But, I've another section `.dynstr` is still differing, which is holding the absolute `rpath` (This is in our Yocto project, there we are passing the `-Clink-args=-Wl,-rpath,$ORIGIN/../lib`, and this `rpath` is getting adding to the `.dynstr` section) . Rust's build system sets `rpath` too by default. (you can use the `rpath` config option to disable this) We should be using a relative rpath though: > `{libdir}` here is `lib` by default and needs to be explicitly changed in `config.toml` if you want a different value. ---
**Author:** ![system](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/system/32/14092_2.png) [@system](https://internals.rust-lang.org/u/system)\ **Post date:** [January 7, 2024, 6:05pm UTC](https://internals.rust-lang.org/t/librustc-driver-so-not-reproducible/19639/20 "2024-01-07T18:05:17Z")
Closed on January 7, 2024, 6:05pm UTC This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.