A plugin system for business applications

[flavius]

Continuing the idea mentioned here Pre-RFC: Runtime reflection

We really need a plugin system to be a valid contender in the area of "business applications".

I've tried to accomplish it with Any and TypeId relentlessly, it's simply not possible without support from the compiler.

I am aware of the performance and safety implications, however:

• this plugin system would have barely any effect on the performance because it would be used at one of the top levels of the architecture of the application using it
• it would be opt-in

What I need to achieve what I can achieve with, say, java modules (module-info.java):

• allow the users of my application to drop .so files in a directory - the plugins
• i'm fine if the plugins cannot be loaded and have to be recompiled because of ABI issues
• ability to inspect what traits are implemented by the plugin, and call them, allowing the plugin to hook itself into the lifecycle of my application; the plugin still has to implement a trait "Plugin", so that I can establish initial communication with it
• the plugins can depend on each-other. TypeId is not enough here, as other plugins are in differend .so files; this is the dependency inversion mentioned in the other thread; yes, it must be ensured that the dependency graph doesn't have cycles; I'm fine with doing that at runtime and showing the user an error with where the cycle is

This would really make rust a good contender for business applications in which companies are willing to pour money and it would help the adoption of rust.

Not having this in rust is to me the biggest showstopper, because rust's type system is amazing.

Speaking of amazing, this is also the reason why a plugin system like described above cannot be hidden behind a C compatible ABI - it would mean giving up one of the main reasons why a company would choose rust in the first place.

If I have to give up traits and generics and all the good stuff, just to have plugins, then rust loses against, let's say java.

You might say "then rust is not the right language for your projects", well yes, currently it isn't.

The question could also be: why don't we want to make rust attractive for more companies who could pour in the money, thus driving rust's adoption forward?

Addendum

I really don't know what it would be best approach in terms of implementation. Probably not reflection, probably some rtti just for this use-case, I really don't know.

But I've been lurking around this community for some years already and I'm sure there are enough bright people who can figure it out, if it's set as a priority.

Also, the plugins don't have to be .so files, they can be in any other format, including a new binary format.

PS: I'm one of those who votes on stackoverflow for "rust most loved", but doesn't say he uses rust professionally.

[josh]

WebAssembly works very well for plugins, and allows you to write the plugins in an arbitrary language. It also lets you sandbox the plugins so they only have access to what you give them access to.

[flavius]

Wouldn't it have the same problems as a C ABI, that I mentioned in the first post, except it's a different ABI?

[H2CO3]

@flavius

The question could also be: why don't we want to make rust attractive for more companies who could pour in the money

Sorry but that really sounds dishonest, it's a hijacking of technical counter-arguments in the direction of "you aren't WILLING to make Rust more attractive to companies".

That is pretty much not the case. People here aren't thinking "hey, we must really stop this guy from making Rust appeal to companies!" In fact I think the sentiment is the opposite, and we generally are happy if some company chooses to use Rust for making something nice.

You should also define what you mean by "business" applications which is a very generic term and I don't see how "business" or "non-business" bears relevance to a technical proposal. Probably you should elaborate on some concrete use cases which are impossible to solve without a reflection mechanism.

[crlf0710]

There's this crate that does exactly what you want i think: https://docs.rs/abi_stable/0.8.3/abi_stable/

But i haven't tried it yet, and no idea what the experience would be.

[Nemo157]

WASM Interface Types, which I believe would be the eventual way to define plugin interfaces, are much more capable than the C ABI. It still looks like it’s a long way away though.

[notriddle]

Yeah, using webassembly for an application plugin system is probably a good idea, but not terribly relevant for this discussion. It's good to be able to sandbox plugins, and good to make plugins architecture-independent. Hello Graviton! Hello macOS 11! Hello IBM i! Hello CloudABI! Let's make plugins that can run under any of these environments!

But building a good extension system requires a lot of design trade-offs that are often different from the ones Rust tried to make. To do it right, Rust would need to replicate the many weird tricks in Swift's ABI, in order to support dynamically linked generic code.

Except that it can't: Swift's ABI sometimes requires objects to be dynamically allocated, and the Rust compiler can't assume the existence of a memory allocator, because not requiring a runtime is Rust's whole purpose in existing.

Existentials are the really tricky case for stack allocations, because they can prevent the caller from knowing the size of the return value before making the call, and that really messes up alloca. So once existentials get involved, alloca goes out the window and actual boxing needs to happen.

I've thought about working around this problem by using stackless coroutines, but that's a leaky abstraction, since there's code that would work with boxing that the stackless coroutine compiler would have to reject.

Edit: I probably should've linked to the updated RFC for returning unsized by stackless coroutines. That post is outdated, but the overall point, that you can't just compile arbitrary Rust code this way and expect it to work, remains.

[comex]

@notriddle

Except that it can't: Swift's ABI sometimes requires objects to be dynamically allocated, and the Rust compiler can't assume the existence of a memory allocator, because not requiring a runtime is Rust's whole purpose in existing.

Existentials are the really tricky case for stack allocations, because they can prevent the caller from knowing the size of the return value before making the call, and that really messes up alloca. So once existentials get involved, alloca goes out the window and actual boxing needs to happen.

I think it might actually be okay. There are two related but distinct scenarios.

Existentials are just trait objects, and Rust obviously already supports those. If you want to return one from a function you indeed must use actual boxing, but you can do that yourself.

Non-monomorphized generics do somewhat resemble trait objects from an implementation perspective. That is, in both of the following examples (assuming the generic one is not monomorphized), the compiler has to generate code for foo without knowing the concrete type implementing Trait at compile time:

fn foo(f: &dyn FnOnce(i32) -> dyn Trait) { let x = f(42); }

fn foo<T: Trait>(f: &dyn FnOnce(i32) -> T) { let x = f(42); }

But there's a key difference. In the trait object version, foo doesn't know the concrete type at runtime, either, until after it executes f. After all, f could return different types depending on its parameter or any other state.

In the generic version, on the other hand, the compiler would add a hidden parameter to foo for a vtable containing information about T, which would be automatically supplied by the caller. So foo does know the concrete type at runtime even before calling f, allowing it to reserve the correct amount of space on the stack using a dynamically-sized alloca.

[notriddle]

That's doable, but now you're subject to the object safety rules.

[comex]

I'm not sure what you mean. Existentials, which are just another name for trait objects, are subject to the object safety rules by definition (although non-monomorphized generics would allow those to be loosened). But you wouldn't be forced to use existentials to get a stable ABI. Can you elaborate on the scenario you're thinking of?

[PoignardAzur]

@notriddle

Except that it can't: Swift's ABI sometimes requires objects to be dynamically allocated,

I'm a little worried that people on this forum seem to refer more and more to this blog post when discussing Swift's ABI, when the author probably didn't intend for it to be a reference on the subject.

@notriddle

and the Rust compiler can't assume the existence of a memory allocator, because not requiring a runtime is Rust's whole purpose in existing.

Is that a fundamental problem? There could simply be two types of Rust dylibs: regular dylibs which need to be passed an allocator at startup, and no_std dylibs which can't create existential types.

[notriddle]

@comex

In the generic version, on the other hand, the compiler would add a hidden parameter to foo for a vtable containing information about T , which would be automatically supplied by the caller. So foo does know the concrete type at runtime even before calling f , allowing it to reserve the correct amount of space on the stack using a dynamically-sized alloca.

Now that I rethink it, it's really size_of that's the problem. As a const function, it cannot be computed from a hidden pointer at runtime.

I mean, I don't know how much dependent types would be needed to make this work. I'm starting to feel a bit out of my league here with some of the subtler aspects of const generics. There's always limits imposed to make type checking itself work, but I haven't truly kept up well enough to know what they are, or if they're enough to make the hidden parameter that you're talking about work.