# A plugin system for business applications
**URL:** https://internals.rust-lang.org/t/a-plugin-system-for-business-applications/12313
**Category:** Uncategorized
**Created:** [May 10, 2020, 6:41am UTC](https://internals.rust-lang.org/t/a-plugin-system-for-business-applications/12313 "2020-05-10T06:41:30Z")
**Posts on this page:** 13
**Page:** 1
### Author:  [@flavius](https://internals.rust-lang.org/u/flavius)
#### Post date: [May 10, 2020, 6:41am UTC](https://internals.rust-lang.org/t/a-plugin-system-for-business-applications/12313/1 "2020-05-10T06:41:30Z")
Continuing the idea mentioned here [Pre-RFC: Runtime reflection](https://internals.rust-lang.org/t/pre-rfc-runtime-reflection/11039/45)
We really need a plugin system to be a valid contender in the area of "business applications".
I've tried to accomplish it with Any and TypeId relentlessly, it's simply not possible without support from the compiler.
I am aware of the performance and safety implications, however:
- this plugin system would have barely any effect on the performance because it would be used at one of the top levels of the architecture of the application using it
- it would be opt-in
What I need to achieve what I can achieve with, say, java modules (module-info.java):
- allow the users of my application to drop `.so` files in a directory - the plugins
- i'm fine if the plugins cannot be loaded and have to be recompiled because of ABI issues
- ability to inspect what traits are implemented by the plugin, and call them, allowing the plugin to hook itself into the lifecycle of my application; the plugin still has to implement a trait "Plugin", so that I can establish initial communication with it
- the plugins can depend on each-other. TypeId is not enough here, as other plugins are in differend `.so` files; this is the dependency inversion mentioned in the other thread; yes, it must be ensured that the dependency graph doesn't have cycles; I'm fine with doing that at runtime and showing the user an error with where the cycle is
This would really make rust a good contender for business applications in which companies are willing to pour money and it would help the adoption of rust.
Not having this in rust is to me the biggest showstopper, because rust's type system is amazing.
Speaking of amazing, this is also the reason why a plugin system like described above cannot be hidden behind a C compatible ABI - it would mean giving up one of the main reasons why a company would choose rust in the first place.
If I have to give up traits and generics and all the good stuff, just to have plugins, then rust loses against, let's say java.
You might say "then rust is not the right language for your projects", well yes, currently it isn't.
The question could also be: why don't we want to make rust attractive for more companies who could pour in the money, thus driving rust's adoption forward?
Addendum
I really don't know what it would be best approach in terms of implementation. Probably not reflection, probably some rtti just for this use-case, I really don't know.
But I've been lurking around this community for some years already and I'm sure there are enough bright people who can figure it out, if it's set as a priority.
Also, the plugins don't have to be `.so` files, they can be in any other format, including a new binary format.
PS: I'm one of those who votes on stackoverflow for "rust most loved", but doesn't say he uses rust professionally.
---
### Author:  [@josh](https://internals.rust-lang.org/u/josh)
#### Post date: [May 10, 2020, 7:07am UTC](https://internals.rust-lang.org/t/a-plugin-system-for-business-applications/12313/2 "2020-05-10T07:07:13Z")
WebAssembly works very well for plugins, and allows you to write the plugins in an arbitrary language. It also lets you sandbox the plugins so they only have access to what you give them access to.
---
### Author:  [@flavius](https://internals.rust-lang.org/u/flavius)
#### Post date: [May 10, 2020, 7:16am UTC](https://internals.rust-lang.org/t/a-plugin-system-for-business-applications/12313/3 "2020-05-10T07:16:13Z")
Wouldn't it have the same problems as a C ABI, that I mentioned in the first post, except it's a different ABI?
---
### Author:  [@H2CO3](https://internals.rust-lang.org/u/H2CO3)
#### Post date: [May 10, 2020, 9:02am UTC](https://internals.rust-lang.org/t/a-plugin-system-for-business-applications/12313/4 "2020-05-10T09:02:34Z")
> [@flavius](#):
>
> The question could also be: why don't we want to make rust attractive for more companies who could pour in the money
Sorry but that really sounds dishonest, it's a hijacking of _technical_ counter-arguments in the direction of "you aren't WILLING to make Rust more attractive to companies".
That is pretty much not the case. People here aren't thinking "hey, we must really stop this guy from making Rust appeal to companies!" In fact I think the sentiment is the opposite, and we generally are happy if some company chooses to use Rust for making something nice.
You should also define what you mean by "business" applications which is a very generic term and I don't see how "business" or "non-business" bears relevance to a technical proposal. Probably you should elaborate on some concrete use cases which are impossible to solve without a reflection mechanism.
---
### Author:  [@crlf0710](https://internals.rust-lang.org/u/crlf0710)
#### Post date: [May 10, 2020, 9:19am UTC](https://internals.rust-lang.org/t/a-plugin-system-for-business-applications/12313/5 "2020-05-10T09:19:20Z")
There's this crate that does exactly what you want i think: [https://docs.rs/abi\_stable/0.8.3/abi\_stable/](https://docs.rs/abi_stable/0.8.3/abi_stable/)
But i haven't tried it yet, and no idea what the experience would be.
---
### Author:  [@Nemo157](https://internals.rust-lang.org/u/Nemo157)
#### Post date: [May 10, 2020, 12:20pm UTC](https://internals.rust-lang.org/t/a-plugin-system-for-business-applications/12313/6 "2020-05-10T12:20:53Z")
[WASM Interface Types](https://github.com/WebAssembly/interface-types/blob/master/proposals/interface-types/Explainer.md), which I believe would be the eventual way to define plugin interfaces, are much more capable than the C ABI. It still looks like it’s a long way away though.
---
### Author:  [@notriddle](https://internals.rust-lang.org/u/notriddle)
#### Post date: [June 25, 2020, 5:53pm UTC](https://internals.rust-lang.org/t/a-plugin-system-for-business-applications/12313/7 "2020-06-25T17:53:52Z")
Yeah, using webassembly for an application plugin system is probably a good idea, but not terribly relevant for this discussion. It's good to be able to sandbox plugins, and good to make plugins architecture-independent. Hello [Graviton](https://aws.amazon.com/ec2/graviton/)! Hello [macOS 11](https://9to5mac.com/2020/06/22/arm-mac-apple/)! Hello [IBM i](https://en.wikipedia.org/wiki/IBM_i)! Hello [CloudABI](https://cloudabi.org/)! Let's make plugins that can run under any of these environments!
But building a good extension system requires a lot of design trade-offs that are often different from the ones Rust tried to make. To do it _right_, Rust would need to replicate the many weird tricks in Swift's ABI, in order to support dynamically linked generic code.
Except that it can't: Swift's ABI [sometimes](https://gankra.github.io/blah/swift-abi/#esoterica) requires objects to be dynamically allocated, and the Rust compiler can't assume the existence of a memory allocator, because not requiring a runtime is Rust's whole purpose in existing.
> Existentials are the really tricky case for stack allocations, because they can prevent the caller from knowing the size of the return value before making the call, and that really messes up alloca. So once existentials get involved, alloca goes out the window and actual boxing needs to happen.
I've thought about working around this problem by [using stackless coroutines](https://internals.rust-lang.org/t/pre-rfc-minimal-flexible-placement-and-returning-unsized-types/10032), but that's a leaky abstraction, since there's code that would work with boxing that the stackless coroutine compiler would have to reject.
Edit: I probably should've linked [to the updated RFC for returning unsized by stackless coroutines](https://github.com/rust-lang/rfcs/pull/2884). That post is outdated, but the overall point, that you can't just compile arbitrary Rust code this way and expect it to work, remains.
---
### Author:  [@comex](https://internals.rust-lang.org/u/comex)
#### Post date: [June 25, 2020, 9:31pm UTC](https://internals.rust-lang.org/t/a-plugin-system-for-business-applications/12313/8 "2020-06-25T21:31:55Z")
> [@notriddle](#):
>
> Except that it can't: Swift's ABI [sometimes](https://gankra.github.io/blah/swift-abi/#esoterica) requires objects to be dynamically allocated, and the Rust compiler can't assume the existence of a memory allocator, because not requiring a runtime is Rust's whole purpose in existing.
>
> > Existentials are the really tricky case for stack allocations, because they can prevent the caller from knowing the size of the return value before making the call, and that really messes up alloca. So once existentials get involved, alloca goes out the window and actual boxing needs to happen.
I think it might actually be okay. There are two related but distinct scenarios.
Existentials are just trait objects, and Rust obviously already supports those. If you want to return one from a function you indeed must use actual boxing, but you can do that yourself.
Non-monomorphized generics do somewhat resemble trait objects from an implementation perspective. That is, in both of the following examples (assuming the generic one is not monomorphized), the compiler has to generate code for `foo` without knowing the concrete type implementing `Trait` at compile time:
```rust
fn foo(f: &dyn FnOnce(i32) -> dyn Trait) { let x = f(42); }
```
```rust
fn foo(f: &dyn FnOnce(i32) -> T) { let x = f(42); }
```
But there's a key difference. In the trait object version, `foo` doesn't know the concrete type at _runtime_, either, until after it executes `f`. After all, `f` could return different types depending on its parameter or any other state.
In the generic version, on the other hand, the compiler would add a hidden parameter to `foo` for a vtable containing information about `T`, which would be automatically supplied by the caller. So `foo` does know the concrete type at runtime even before calling `f`, allowing it to reserve the correct amount of space on the stack using a dynamically-sized alloca.
---
### Author:  [@notriddle](https://internals.rust-lang.org/u/notriddle)
#### Post date: [June 25, 2020, 10:17pm UTC](https://internals.rust-lang.org/t/a-plugin-system-for-business-applications/12313/9 "2020-06-25T22:17:12Z")
That's doable, but now you're subject to the object safety rules.
---
### Author:  [@comex](https://internals.rust-lang.org/u/comex)
#### Post date: [June 26, 2020, 4:03am UTC](https://internals.rust-lang.org/t/a-plugin-system-for-business-applications/12313/10 "2020-06-26T04:03:12Z")
I'm not sure what you mean. Existentials, which are just another name for trait objects, are subject to the object safety rules by definition (although non-monomorphized generics would allow those to be loosened). But you wouldn't be forced to use existentials to get a stable ABI. Can you elaborate on the scenario you're thinking of?
---
### Author:  [@PoignardAzur](https://internals.rust-lang.org/u/PoignardAzur)
#### Post date: [June 26, 2020, 7:23am UTC](https://internals.rust-lang.org/t/a-plugin-system-for-business-applications/12313/11 "2020-06-26T07:23:28Z")
> [@notriddle](#):
>
> Except that it can't: Swift's ABI [sometimes](https://gankra.github.io/blah/swift-abi/#esoterica) requires objects to be dynamically allocated,
I'm a little worried that people on this forum seem to refer more and more to this blog post when discussing Swift's ABI, when the author probably didn't intend for it to be a reference on the subject.
> [@notriddle](#):
>
> and the Rust compiler can't assume the existence of a memory allocator, because not requiring a runtime is Rust's whole purpose in existing.
Is that a fundamental problem? There could simply be two types of Rust dylibs: regular dylibs which need to be passed an allocator at startup, and no\_std dylibs which can't create existential types.
---
### Author:  [@notriddle](https://internals.rust-lang.org/u/notriddle)
#### Post date: [June 26, 2020, 2:33pm UTC](https://internals.rust-lang.org/t/a-plugin-system-for-business-applications/12313/13 "2020-06-26T14:33:56Z")
> [@comex](#):
>
> In the generic version, on the other hand, the compiler would add a hidden parameter to `foo` for a vtable containing information about `T` , which would be automatically supplied by the caller. So `foo` does know the concrete type at runtime even before calling `f` , allowing it to reserve the correct amount of space on the stack using a dynamically-sized alloca.
Now that I rethink it, it's really [`size_of`](https://doc.rust-lang.org/std/mem/fn.size_of.html) that's the problem. As a const function, it cannot be computed from a hidden pointer at runtime.
I mean, I don't know how much dependent types would be needed to make this work. I'm starting to feel a bit out of my league here with some of the subtler aspects of const generics. There's always limits imposed to make type checking itself work, but I haven't truly kept up well enough to know what they are, or if they're enough to make the hidden parameter that you're talking about work.
---
### Author:  [@system](https://internals.rust-lang.org/u/system)
#### Post date: [September 24, 2020, 2:34pm UTC](https://internals.rust-lang.org/t/a-plugin-system-for-business-applications/12313/14 "2020-09-24T14:34:10Z")
This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.