Hi folks,
This post has a few aspects to it. It is partly a “bug report”, partly a question about whether it is even a bug or just not-yet-specified behavior (or “it’s specified as your_fault_fool”), and partly a solicitation for similarly minded folks.
I’m currently doing a bit of work where I believe that what I want is Rust on Rust FFI, to support some REPL-like behavior for differential dataflow. The current mock-up is a static library and server binary defining some common types, with the intent that you write shared libraries using these types to interact with the underlying timely dataflow runtime and then the server loads and unloads them for you interactively.
This type of project doesn’t seem wildly unexpected. In managed environments this is often pretty easy to do, dynamically loading code and interoperating safely, and it seems like Rust is pretty close to being able to do it with some confidence.
There are a bunch of potentially naughty aspects to this, some of which I know about and probably at least as many I don’t know about. For example:
-
Passing around owned data with any sort of backing memory is perilous, as by default the allocators for dynamic libraries and binary are different (resp: the system allocator and jemalloc). I’ve swapped in the system allocator for the binary, and am working under the (possibly flawed) assumption that if the same allocator is used then either piece of code can allocate/deallocate without risking explosions.
-
Passing around data of any sort relies on some assumptions about data layouts, and my understanding from
#rust-internals
is that there are sane assumptions to make if you only use types drawn in from common binary sources (i.e. a binary locks down the layout of types it exposes). Roughly, for the same reason that coherence is required (different crates independently using the same types) there is not much flexibility in re-laying things out. I’m still not 100% on what the guarantee is (or if it is a guarantee, vs a “for the foreseeable future”). -
If you ever unload libraries you’d best be sure you don’t need that code ever again. Like, if you return a
Box<Any>
back to the server, say, and try to use it. >.<
To the extent that there are other folks interested in understanding this, or in articulating some guarantees (perhaps outside “stability guarantees”, just with the goal of putting words to intent), I’d love to hear from you.
The “bug report” part of this is that I currently get segfaults (at runtime) when using workspaces to link the building of the library, server binary, and example dylibs together. If I unlink them and build separately, the segfaults seem to go away (though difficult to know if they are just deferred, because I am not sure whether what I am doing is intended to work or not). My guess is that workspaces are meant to be very similar to building each of the projects, and if there is a different result that is interesting, but if it is segfaulting then it is probably UB and so maybe not out of spec in any way.