Hi all. First off I would like to preface this with the fact that I am not a compiler developer so everything below might completely impossible. But I figured I would try anyway. I also realise that there are at least a couple of pre-existing projects looking into the topic of dynamic libraries and this might not be needed, but hopefully this strikes a balance between not having any kind of stable ABI and having the same fixed ABI for the rest of eternity.
I took my inspiration from how Windows handles compiling DLLs. For those who’ve never compiled a shared library on Windows what you get when you compile the library is the .dll file and a secondary .lib which contains effectively a stub of all the publicly accessible symbols in your .dll. Then at linking time you link to the .lib which tells your application how to actually access and use the symbol.
So where does Rust fit in? Well I think the same pattern could be applied to Rust as well when compiling dynamic libraries. Only instead of getting a platform specific .lib file you get something like a .rimport file or something. similar. The goal would be that during compilation of the library every publicly accessible symbol you have would be exported into the .rimport file along with the ABI that it was compiled with. No fixed ABI here, just the one that toolchain used at the time it was compiled.
The goal would be that once you have compiled your library someone else can link against it and point the compiler at your .rimport file to get all the required API and ABI information. Then during compilation of the consumer application anytime the compiler comes across a reference to a symbol from the library rather than using it’s own local ABI it uses the ABI of the compiled library in the .rimport file. To define the ABI I would recommend that, just like Rust Editions, you have ABI Editions. Something that is concrete for only the exported symbols and everything internal to the library would still use Rust’s current ABI system. This would in theory allow updated dynamic libraries to be drop in replacements as long as they are built to the same Edition of the ABI and their API/structure hasn’t changed. Then in the future shoud a new version of the ABI be released it becomes the user’s choice which to use and the compiler would still know how to handle it.
I can also see this sort of system being useful outside of Rust by providing a standardised format that other languages could use to import Rust code into their languages. Python springs to mind.
There’s about a thousand different ways I could extend this but will stop here for now. Would love to hear some thoughts from people who actually do compiler/linker developement. Specifically if something like this is even possible.
Good news: these files already exist. They are called metadata files, with extension .rmeta. They contain exactly the information that the compiler of some dependent code needs in order to compile against the library.[1]
So, you don’t need to invent a new file format; you “just” need:
sufficient ABI stability/versioning, and
a way to tell Cargo to accept a pre-existing .rmeta rather than compiling all dependencies from source.
This doesn't work very well the moment you have some kind of indirection. If your function accepts a &Foo then the ABI of the function cannot dictacte how to pass the Foo behind the reference. This in turns means that either the caller's Foo is the same as the dynamic library's or they are different Foos.
Now imagine if Foo is a stdlib type. The suddently you and the dynamic library are basically speaking two different dialects of Rust!
This is also missing some safety measures at linking time. Are you just trusting that the library loaded at runtime will be ABI-compatible with the .lib provided? What about ABI-compatible changes though?
Note that even if the Rust-level ABI is the same that doesn't mean a type cannot change (and in turn change its ABI) across different versions of a library!
Finally, I assume this doesn't aim to handle generics. This should be noted somewhere since generics always come up when talking about dynamic loading.
Good news: these files already exist. They are called metadata files, with extension .rmeta. They contain exactly the information that the compiler of some dependent code needs in order to compile against the library.
Hey that's cool to know. I assume that maybe some additional information would be needed. But in either case having these files in place is a good start
This doesn't work very well the moment you have some kind of indirection. If your function accepts a &Foo then the ABI of the function cannot dictacte how to pass the Foo behind the reference. This in turns means that either the caller's Foo is the same as the dynamic library's or they are different Foos.
Not sure I follow, if the compiler has the ABI of Foo in the .rimport file and it also has the ABI of the struct at the callsite then surely it could compare the two and error out if they don't match? Isn't this how Stabby does it's compatability checks between libraries?
Sure, but then changing the internal representation of a type becomes a breaking change. Linking to a dynamic library would become a matter of using the correct compiler version and dependencies versions, and linking to multiple dynamic libraries at the same time could be impossible due to conflicting requirements.
ps: I suggest you to use the native quote feature instead of >, that way the quoted people will see a notification.