What is the current state of sharing a Rust lib crate between executables in a similar way to dynamic link libraries?

I have a large battery-included library (lib crate) and a bunch of small executables that use it, with a relationship a bit similar to libav and ffmpeg. ffmpeg can be built as "shared", which means that the 3 executables (ffmpeg, ffplay, ffprobe) can share the libav dynamic link libraries, which significantly reduce the total binary size compared to "static" build.

So is there a way to do something like this in Rust? The below is how I imagine this would ideally be implemented, which I think may be feasible but may be difficult.

First, compile the lib crate into some form of binary. (It maybe not necessarily the target platform's dynamic link library format. I think it can be some Rust defined format, is rlib possible?) Then declare the expected path to the lib crate binary somewhere (Cargo.toml or code) in the bin crate (of course with the source dependency in Cargo.toml too), and use the various things (functions, constants and maybe even macro_rules) provided by the lib crate with module structure kept, just as a normal static-linked lib crate. When executing, the bin crate executable searches for the lib crate binary, verifies the version of rustc (to prevent ABI incompatibility) and the lib crate, then starts normally.

After searching a lot, I found that most of the description and discussion of how to do this in Rust seemed vague (in my view). And the only few solutions that seemed to work try to compile the lib crate into the target platform's dynamic link library format, and then call it as a normal dynamic link library, whether using C ABI or Rust ABI, which is obviously troublesome and restrictive and more unsafe. And most of them were released/updated two or three years ago. There seems to be an issue of cargo that is still open, but it seems to be no progress. So I'd like to know the current status of doing this.

1 Like

While you could technically get away with using a shared .rlib for a tightly-controlled set of executables, it's not a supported use-case, and the tooling is not prepared for this. Rust is not ready to support such use (that would require having a stable ABI and figure out what to do about generics). It may take a long time before it's supported. In the meantime if you run into any problems with a DIY approach around rlib, you won't be able to count on any help with it, because it's not supported that way.

You need to export a C ABI from a dynamic library (set crate-type to cdylib or even use cargo-c). It is a poor solution. It is unsafe. It is system-dependent. It is a pain, but Rust still doesn't have anything better. There is a crate that helps translate Rust types into C types to make it less tedious:

1 Like

As an alternative, would it work for you to use the "multicall binary" pattern, as used by BusyBox and uutils, in which the various commands are merely filesystem links to an executable that chooses a function to execute based on the name by which it is invoked?

12 Likes

The reference seems to imply that you can use crate-type = ["dylib"] (instead of cdylib), but it's not clear what restrictions this has from that page. In particular, I suspect you would have trouble using generics, or bring able to share standard library types across the boundary which is fairly restrictive.

1 Like

Do you mean importing consts (which the whole point is having them available at compile time) and macro_rules (which must be expanded while still parsing, even before type checking) from a dynamically linked library? If so then you're pretty much asking to compile a rust library on the fly, deferring any compilation error at runtime.

Generally the only things you can move to a shared library are non-generic functions and statics. This is also still a pain to do because it's unsafe and you need to use C FFI, but it's doable just like any C shared library did it until now.

You can use generics just fine. They will just be duplicated across the boundary. The only real restrictions are that LTO isn't supported (see [PERF] Enable LTO for rustc_driver.so by bjorn3 · Pull Request #101403 · rust-lang/rust · GitHub for the changes necessary to support it) and that you will get an error if any crate ends up being linked into multiple dylibs and you try to link against both dylibs. Another thing is that you must ensure that the dylib remains the same between compile time and runtime. You have to recompile all dependent crates if you want to update a rust dylib.

3 Likes

Wow, that's better than I hoped: sounds like it should match the OP's situation, at least, even if it's a bit of a rarer use-case.

In fact, my intention is that from the bin crate writer's point of view, there is no difference between using this and normal static-linked lib crates. The bin crate must still be compiled with a reference to the source code of the lib crate in order to finish all the const inlining, macro expansion, generic casting, etc. at compile time. The "some compiled product of lib crate" that is dynamic linked to at runtime should actually only contain functions and statics.

In fact, there is a key demand not mentioned: The number of executables may increase dynamically, but it is guaranteed that they all depend on the same version of the lib crate and complied by the same version of rustc. In other words these executables are something more like plugins. So neither multicall binary nor subcommands is feasible.

At this point in time, I don't think that there is a good way of doing precisely what you want in Rust. Given your requirements, you might want to look into some kind of RPC solution. Maybe one of these would work for you?

I haven't used any of these and can't make any recommendations, so if you choose to go this route you will need to test and evaluate them on your own.