Hi Everyone, I work on the NVIDIA Linux GPU driver and we have discovered a bad interaction with certain rust applications. It's a bit complicated to explain, so please bear with me.
A number of rust crates providing bindings to system libraries have the option to be linked statically into the final application binary. The naming convention for this seems to be "vendored". When this is done, it would appear that all externally visible symbols from the library remain in the global symbol table of the final executable.
This is problematic.
Consider the following scenario, based on a real-world case...
A rust application includes the "dbus" crate as a dependency and enables the "vendored" feature mentioned above, meaning libdbus will be statically linked. Inspecting the application binary, we see dozens of dbus functions in its symbol table. One if these is dbus_threads_init, a function that, among a few other things, initializes some global mutexes that are use to ensure thread-safety.
This application also dynamically links against webkit2gtk. Like many web engines, WebKit uses OpenGL for hardware accelerated compositing, therefore when the application runs and loads libwebkit2gtk, that loads the system OpenGL implementation as a dependency.
If it's a recent NVIDIA implementation, it will dlopen libdbus (*) and look up some function symbols using dlsym. This loads the system copy of libdbus into memory. But recall that the application binary has its own, statically linked, copy of libdbus embedded directly within it, and that all of its non-static (as in the C keyword) symbols are in the global symbol table.
What happens is that the entry-point function symbols returned by dlsym will be in the dynamically loaded copy of libdbus. However, any internal function calls will jump to the corresponding function in the embedded copy. Interestingly, the same is not true for global variable references... for some reason related to LTO.
That's the source of the trouble. The first entry point we call (dbus_bus_get) will, in turn, call that dbus_threads_init function I mentioned above. But it will call the function from the embedded copy of the library, and that function will initialize the embedded copies of those global variables. However, after it returns, we're back in the dynamically loaded copy and our global variables are still uninitialized. Eventually this causes a segfault.
"Aha!" you may say, "this is what RTLD_DEEPBIND is for!". Well, yes, as I described things above you'd be right. But I left out one caveat for brevity. What actually happens is that libwebkit2gtk loads libdbus as a dependency (for its own reasons), and that's when the problematic symbol-binding happens. When our OpenGL implementation later calls dlopen on the same library it's a no-op. Even if we pass RTLD_DEEPBIND, the damage has already been done.
So there is it. Hopefully that was as least semi-coherent. My question to the rust internals community is whether this behavior - putting symbols from statically linked libraries in the global symbol table of the final binary - can be changed. It seems like it ought not to be necessary... or is it for some reason?
Thanks! Erik
(*) why does an OpenGL driver need dbus? It's used by a new-ish power-management feature for laptops, Chapter 23. Dynamic Boost on Linux