Dynamically Linking Rust Crates to Rust Crates

[m4b]

@hanna-kruppe

Nobody should lock themselves up in a room for six weeks and emerge with a finished proposal for any big topic.

@Centril

You’d have to actively propose a stable ABI for us to give justifications on an RFC.

Isn’t this explicitly asking someone to emerge, whether locked up or not, with a stable ABI documentation?

[m4b]

This entire discussion is getting derailed. A Lang team member posted that they are not considering abi stability, perhaps ever. I asked for clarification. I received none. Not sure what’s difficult here.

[Centril]

@m4b

Isn’t this explicitly asking someone to emerge, whether locked up or not, with a stable ABI documentation?

The language team as a group does generally not make judgements on half-baked drafts. However, language team members (e.g. myself and @scottmcm) individually do frequently provide such reasoning. This is a distinction about the team's collective responsibilities and the opinions of individual team members.

From my perspective:

1. We already have several stable opt-in ABIs / layouts.

2. A stable ABI is a hazard in terms of past and future compiler optimizations. Once you stabilize the ABI, which may be convenient for current hardware, you have to live with it forever (unless the ABI is versioned, in which case you add a bunch of complexity).

3. A stable ABI is a hazard in terms of future language design. For example, if we specify that the layout of (A, B, C) is equivalent to struct Foo(A, B, C); then we cannot provide certain language features in certain ways for tuple generics.

For these reasons, I generally strongly prefer opt-in mechanisms such as repr(C), repr(transparent), etc. As @hanna-kruppe notes, it is far more productive to discuss small proposals to stabilize certain parts of "the ABI" where we can consider the specific risks and benefits of such proposals in detail rather than doing whole-sale stabilization of the whole ABI, name mangling, etc.

[m4b]

Thank you for your response

[Centril]

@zrk

I wonder if we could allow applying #[repr(C)] attribute to any existing type such that #[repr(_)] T would reexport any T while deeply changing its internal representation to #[repr(C)] . This is different from applying #[repr(C)] to a newtype #[repr(C)] struct Foo(Bar); as, if I’m not mistaken, in the latter struct declaration we don’t change the representation of Bar inside of Foo .

This would allow us to e.g. declare function like the following:

pub fn foo(x: #[repr(C)] Vec<u8>) -> #[repr(C)] Bar { /* ... */ }

foo is a publicly-exported function that takes a Vec<u8> changed to have a C repr and returns a Bar custom defined library type changed to have C repr.

This is an intriguing idea that I've thought a bit about before. For now, I have a proposal that would add the syntactic support for attributes on types:

RFC: #[attribute]s galore

This would facilitate moving towards #[repr(C)] Foo if we wanted to.

What gives me pause is that for soundness purposes, #[repr(C)] Foo, should, as far as I can tell, be a nominally distinct type from Foo. What does this mean for all the operations and implementations that exist for Foo?

@zrk

We could allow this with a special mem function to_c_repr that would basically have the following signature:

fn to_c_repr<T>(this: T) -> #[repr(C)] T { /* would need to be implemented using compiler intrisics, I presume */ }

It would be good to provide a basic informal argument for why this function is sound, particularly if we extend this to other repr(..) modifiers such as alignment.

@zrk

since we would have to recursively call to_c_repr on all fields and subfields of T .

This surprises me because I would expect that struct Foo(Bar, Baz); + #[repr(C)] Foo would be equivalent to the nominal type #[repr(C)] struct FooC(Bar, Baz);.

[zicklag]

@Eh2406

Most of the compile time benefits can more easily be gotten by using one shared target directory for all projects on the same computer. You can experiment with this behavior on stable by setting the CARGO_TARGET_DIR env.

That's very cool! I've set my env and I'll try it next time I cargo build something.

---

@matt1985

I am not sure why this is a problem,regular libraries don’t have access to the API of their dependent crates.

It is not a requirement of the app having to get full API access to their dependent crates, it is a matter of the plugin getting full access to the app crate. I walked through the workflow that I was imagining in a tutorial that I made.

Overall the workflow is:

1. App starts up.
2. App loads plugin by running a specific, pre-deterimined function on the plugin dylib using rust_dlopen.
   • Other callbacks can be triggered by the app in the same way.
   • This does represent an interface that is pre-determined by a standard.
3. The plugin, when loaded, is dynamically linked to the app and can use the entire API of the app.
   • The plugin can import any structs required to properly reply to callbacks and can call any functions exposed in the public API of the app.
   • This does not need to have any interface defined by interface crates of any sort. It can use app_crate::TypeName just like you would if the app crate was statically linked to the plugin.

The problem is that the tutorial used dylib. I made the tutorial before I knew what the different lib types meant.

@matt1985

The way a plugin would communicate with an application is by having some interface crate that defines the public API of the application(to dynamic libraries). I will later write an example of this using #[sabi_trait] derived trait objects,once I finish the first version of them.

That does sound useful and may cover the use-case well enough, but the original goal was avoid having to define interfaces differently than any other crate ( except for the initial plugin registration and the callbacks ).

---

Overall, the prospect of enabling dynamic linking in Rust was quite a bit of a bigger deal than I first realized. My pursuit of the subject was because it would help me realize a simple way to dynamically load Rust code without hardly changing the workflow that was already standard for writing and using Rust crates.

When it comes to the technicalities of what it would take to achieve a stable ABI, most of this stuff is a bit over my head as I just started looking into this at a lower level. At this point I'm not in a position to say whether or not I think we should seek to provide a stable ABI for Rust or not. For now I'll lead that to people more qualified.

As for my specific use-case, I am going to pursue Amethyst's scripting RFC like @CAD97 mentioned above. That will cover my specific use-case for the time being so nobody has to try and figure out a way to satisfy that on this tread anymore.

---

@zrk

Considering the “rust dynamic library” use case, it seems like the goal is to be able to define a “library interface” consisting of public functions and public types exported by the library.

That is a lot like what I was thinking when I was trying to figure out if there was a way to automatically expose a C ABI for public items. As for the technical explanation of how it would work, I don't understand that stuff yet, but I like the high-level idea.

If there is still discussion to be made on techniques for handling dynamic libraries in Rust you may continue, but at this point I'm satisfied with finding another solution. Thanks all for your help!

[matt1985]

I am not seeing what makes abi_stable unable to do what you do in the tutorial,aside from a lack of trait object support(which I am currently working on).

I also don’t know how anyone could do significantly better than what abi_stable does without having direct support from the Rust compiler.If you know better(or even if you have a general idea of alternative designs),you should create an issue explaining what you believe is a better solution that can be done today.

[zicklag]

@matt1985

I am not seeing what makes abi_stable unable to do what you do in the tutorial ,aside from a lack of trait object support(which I am currently working on).

I may have misunderstood what exactly was possible with the abi_stable crate. I'll look more into it and test it out if I run into the need again. It does look like you've got a very cool design for your crate; dynamic linking was just my first choice for an attempted solution because, in my head at least, it was the most language integrated way to do it. Except for the fact that it isn't actually supported.

@matt1985

I also don’t know how anyone could do significantly better than what abi_stable does without having direct support from the Rust compiler.

I don't expect you to be able to do any better at all without support from the compiler, which is why I opened the topic.

[zrk]

Yes, the idea I described would require the RFC you linked to be accepted, or we could use a specific "magic" type wrapper ReprC<T>, but I find the syntax much less convenient.

To elaborate on the idea, I propose to add the ability to "deeply" convert a type representation to repr(C). That is, given the following structs:

struct Foo(String, Vec<u8>);
struct Bar(Option<Foo>);
struct Baz {
    x: Foo,
    y; Bar
}

that all have the default repr(rust) representation (so we can't say anything of their layout), provide a way to "convert" e.g. the type Baz to a C repr when it appears as argument or return value of a function:

fn f(baz : #[repr(C)] Baz) {} 

where #[repr(C)] Baz has a C layout, as if it was another struct BazReprC with the following definition:

#[repr(C)]
struct BazReprC {
    x: #[repr(C)] Foo, // note that the attribute is reapplied recursively here
    y: #[repr(C)] Bar,
}

This might be a little surprising to "deeply" modify the representation of a struct with this syntax, considering that #[repr(C)] struct Foo only affects Foo in a shallow form (that is, the fields of Foo keep their default representation), but I believe such "deep" representation change is necessary to actually be sound when using these types from e.g. a dynamically linked library, otherwise you wouldn't be able to use a Vec<String> using this method, because it would make the fields of the Vec accessible, but these would still return references to String (not #[repr(C)] String) with an unspecified layout. This would allow to access any field or subfield of Baz, as long as it is from a function that takes a #[repr(C)], even if Baz itself was defined with the default #[repr(rust)] representation.

Now, to the difficult parts about the idea...

@Centril

What gives me pause is that for soundness purposes, #[repr(C)] Foo , should, as far as I can tell, be a nominally distinct type from Foo . What does this mean for all the operations and implementations that exist for Foo ?

Yes, #[repr(C)] Foo and Foo would be different types, with possibly different layouts, and should be treated as different types in all contexts, such as generating different monomorphizations and a different set of operations (functions, methods) for Foo and #[repr(C)] Foo. Under my proposal, #[repr(C)] Foo would expose the same operations as Foo. For generics, any generic parameter T would be replaced by #[repr(C)] T in all definitions. #[repr(C)] Foo would additionally expose more operations than Foo: functions that take #[repr(C)] Foo, plus methods that take self: #[repr(C)] Self.

I believe this should be sound, according to the following intuition: today, the #[repr(rust)] representation is not specified, meaning that programs shouldn't rely on specifics of the current representation, and that the current representation could change in the future without breaking existing programs. In particular, the representation could change to become #[repr(C)]. #[repr(C)] Foo is merely a syntactic way to opt into this particular representation for parts of the program.

However, the current proposal does a bit more than that, it proposes to be able to convert at runtime from the default representation (Foo) to the repr(C) representation #[repr(C)] Foo. On to this point:

@Centril

It would be good to provide a basic informal argument for why this function is sound, particularly if we extend this to other repr(..) modifiers such as alignment.

Unfortunately, I cannot find an argument as to why this function would be sound for any arbitrary T. Specifically, things begin to fall apart in the presence of references or pointers in T. Under this "deep" #[repr(C)] proposal, every &'a T in #[repr(C)] Foo should become &'a [#repr(C)] T, but I don't know to which object it could point...

@Centril

This surprises me because I would expect that struct Foo(Bar, Baz); + #[repr(C)] Foo would be equivalent to the nominal type #[repr(C)] struct FooC(Bar, Baz); .

I can understand that. Perhaps the #[repr(C)] Foo syntax is a bit misleading, since, as explained in the beginning of this current message, it would rather be equivalent to #[repr(C)] struct FooC(#[repr(C)] Bar, #[repr(C)] Baz);, ie

#[repr(C)] struct BarC { /* still #[repr(C)] recursively */ }; 
#[repr(C)] struct BazC { /*still #[repr(C)] recursively */ }; 
#[repr(C)] struct FooC(BarC, BazC);

Maybe we should call this #[rrepr(C)] Foo (recursive repr) instead?

However this seems unsound in presence of indirection...

[Centril]

@zrk

but I believe such “deep” representation change is necessary to actually be sound when using these types from e.g. a dynamically linked library,

I was thinking of #[repr(C)] Foo mostly as a convenience for FFI. and not specific to dynamic linking overall but that no implementations and functions would be provided; you'd need to rewrite those.

@zrk

Specifically, things begin to fall apart in the presence of references or pointers in T . Under this “deep” #[repr©] proposal, every &'a T in #[repr(C)] Foo should become &'a [#repr(C)] T , but I don’t know to which object it could point…

This holds for type constructors more generally because this is a question of subtyping. More specifically, to encode:

fn to_ref_c_repr<'a, T: 'a>(this: &'a T) -> &'a #[repr(C)] T { magic }

we need to ensure that T <: #[repr(C)] T.

Otherwise we cannot claim that Δ ⊢ τ <: σ in:

Δ ⊢ τ: 'a   Δ ⊢ σ: 'b
Δ ⊢ 'a: 'b  Δ ⊢ τ <: σ
----------------------
Δ ⊢ &'a τ <: &'b σ

[comex]

@zicklag

There is a crate, abi_stable_crates, that allows dynamically linking other Rust crates like a kind of plugin, but it requires manually specifying the interface over which the crates can communicate and does not allow the plugin crate full access to the public API of the application crate. [..] Dynamically linking Rust crates to other Rust crates over different versions of Rust is unsupported because of the unstable Rust ABI.

I think there's a bit of a contradiction here. I assume you don't just want to have crate A dynamically load crate B, where crate B was previously compiled against the exact same version of crate A, but for some reason using a different version of Rust. Rather, you want to be able to make changes to crate A and continue using the previously compiled crate B. In other words, you need crate A itself to have a stable ABI.

But even in C or C++ where the language has a stable ABI, keeping a library's ABI stable is a significant amount of work beyond just keeping the API stable. It requires careful consideration of the entire interface – deciding which things, such as structure layouts or function definitions, are allowed to change (and hence cannot be inlined or directly exposed to clients), and which are set in stone (and can). That's not such a far cry from using abi_stable in Rust.

In fact, I think there's room to do better than C and C++ in that regard (for example, making ABI breakage result in a compile error rather than mysterious segfaults), and Swift is taking interesting steps in that direction – but it can only go so far. There are often fundamental tradeoffs between performance and flexibility: to preserve the ability to change implementation details of a library while keeping its ABI stable, you need to use additional indirection across the ABI boundary, which in turn has a performance cost.

That said, libraries with stable ABIs are common in both C and C++, and the fact that they're so common despite the amount of pain involved – especially in C++, where language deficiencies force users to resort to obnoxious hacks to hide implementation details – indicates that users derive great value from them. So I agree Rust needs a better story here, even though it may – and should – look very different from how C++ approaches it.

[djc]

Might also be interesting to think about versioning the ABI, for example by tying an ABI to an edition?

[HadrienG]

Yes! I’ve seen proposals along the lines of #[repr(Rust-v1)] in another topic, and I think that could be a very interesting track of thought for those sorts of use cases.

[zicklag]

@comex

I think there’s a bit of a contradiction here. I assume you don’t just want to have crate A dynamically load crate B, where crate B was previously compiled against the exact same version of crate A, but for some reason using a different version of Rust. Rather, you want to be able to make changes to crate A and continue using the previously compiled crate B. In other words, you need crate A itself to have a stable ABI.

Yes, in the case that the API ( and therefor ABI ) of crate A changes, and it tries to load crate B which was built against a previous version of crate A, you would get a runtime error, but if you had crate B specify which versions of crate A it was compatible with, then crate A would be able to detect that crate B is incompatible when it calls crate B's plugin register callback and gets all of the metadata about the plugin in crate B. This would happen before any code from crate B is run, so that crate A could choose not to load it and get away without a runtime error.

[kornel]

@m4b

swift just stabilized theirs and is already reaping the benefits, swift system libraries, the swift runtime, swift UI libraries, all dynamically linked and backwards abi compatible.

I'd love to use Swift ABI from Rust. extern "C" as the lowest common denominator is too low for Rust, but maybe there's a useful subset of Rust that can be mapped to Swift's ABI to provide useful interfaces for Rust and Swift?

[zicklag]

This is probably not practical, but it would be awesome if you could create compiler plugins that could allow you to build your own ABI, or otherwise have some way that a compiler plugin could hook into the compilation process to provide programatic bindings to a custom ABI. Then anybody could make their own stable ABI or make Rust compatible with another ABI such as Swift.

[m4b]

This might be an interesting way to:

1. Prototype an abi
2. Experiment with different abi implementations
3. Scientifically measure the supposed downsides of: a. Having a stable abi b. And also comparing one abis downsides to another abi

[m4b]

I’ve been thinking about a rust swift abi bridge after reading their abi proposal. I would love to work on it, I think it would be massively cool (imagine writing the brand new swift ui in rust ), but I don’t see myself having time in the near future so I’m hoping some adventurous soul will take up the mantle!

So my first thought, to be honest I think a lot of it could be implemented with a procedural macro, eg #[derive(Swift)]

I implemented a basically seamless ffi between OCaml and rust here using regular macros! https://github.com/m4b/raml

Someone started implementing a coq <-> rust proc macro too; and these are in some sense more complicated than Swift’s abi because you need to interact with the gc, link against (and use) static symbols out of the runtime, etc.

The ABI of swift afaics is programmatic; you provide witness tables and bla bla for various generics implementations, which should all be codegennable. Once the abi qua api is understood I think many parts will just fall out.

Probably the only hard part will be the calling convention if it changed, but that should also technically be implementable if necessary via a proxy assembly stub to forward arguments correctly.

I’m sure I’m glossing over important details (the devil is always in the details) but I didn’t see anything particular difficult from a programmatic point of view; at the end of the day there are just calling conventions struct layouts and runtime apis that manage the bridge. We just need some people with some time and I can see a really beautiful and seamless swift abi bridge quite naturally forming

[glandium]

Another aspect of the problem is build support from cargo: currently, if you want to have rust crate A expose a stable C ABI, and rust crate B use it, the only way you can actually do that is to build rust crate A (as in, invoke cargo build) from the build.rs of rust crate B, which is suboptimal.

[proc]

For now you could take a look into https://github.com/mozilla/sccache to get faster compilation times across different projects. I’ve setup my user ENV such that all rust builds use sccache and limited the cache size to 1GiB.