Better C++ interoperability

[alexcrichton]

My own personal feelings hare that I’m all for boosting Rust’s interop with other languages, systems, runtimes, etc, but also keeping in mind the impact on the compiler and language itself. For example developing this as a codegen plugin seems like a great idea as it allows fast iteration outside of the compiler while also keeping a clear list of what’s needed in the compiler to support this kind of plugin.

I don’t have too many opinions on early vs late specialization, but I would hazard a guess and say that the “infectious nature” may not matter too much given Rust’s conventional compilation model. If the standard library required this, for example, that would be a non-starter because that binary is being distributed all over and would impose extra requirements. Most builds are through cargo, however, where binaries are never shipped anywhere and imposing requirements on a downstream crate basically just means that in a few seconds you’ll be required to run the same C++ compiler again. Basically because compiles are relatively self-contained I’d guess that it may not be too much of an issue.

On a related note, the idea of cross-language inlining has definitely been brought up before and is certainly a desired topic! I think Servo wanted it awhile ago for small function calls into SpiderMonkey and we’ve wanted it in the past for small calls into LLVM as well. If that kind of support could be bolstered (or brought into existence) as part of this work, that’d be awesome!

[matklad]

I suppose a lot can be learned from SWIG experience: http://www.swig.org/Doc3.0/SWIGDocumentation.html#SWIGPlus

Because of its complexity and the fact that C++ can be difficult to integrate with itself let alone other languages, SWIG only provides support for a subset of C++ features. Fortunately, this is now a rather large subset.

[nikomatsakis]

Hmm. So, I agree that improving our C++ interop story would be great. The hard part is figuring out just what set of features etc we can support and which we can’t. I had been thinking more of supporting some means I had hoped at some point to read more into what D does – I believe they have defined a subset of C++ where interop works pretty smoothly, and perhaps SWIG would be a good place to look as well.

As far as supporting templates go, I don’t see any way around the “infectious plugin” problem, except of course baking it into rustc (the ultimate infectious plugin). It’s worth keeping in mind that the plugin is only infectious if you fail to encapsulate the C++ library, as well. Hard to say what will be the common case.

I’m curious to dig into this plugin a bit more. This is a good example of a plugin that goes beyond syntax extensions in terms of its requirements. What does it need precisely from rustc? It clearly cannot run as a pre-pass, unless you declare in your source a kind of list of all the instantiations you require. We certainly could just standardize a means for rustc to inform the plugin of what is needed. Alternatively, I wonder though if it could run without deep integration with rustc as a post-pass by just scraping off unresolved symbols from a resulting artifact or something like that. I guess I should look at your example prototypes.

[jan_hudec]

####Early (explicit) specialization

…

The downside is, of course, that the set of specializations is closed. If author of the Rust crate wrapping that C++ library had neglected to provide a specialization for your favorite type, well, tough luck…

There is kind of third way: specialize to a trait type, possibly with a bit of dynamic glue like Any.

This would be similar to how Vala handles its generics. In Vala a generic gets a type descriptor that has a pointer to ref/copy function (corresponding to Rust’s Clone::clone) and unref/delete function (corresponding to Rust’s Drop::drop) and pointers to any method functions the template refers to. And values of the parameter type are handled by generic void *. Well, that is pretty much the same thing as Rust’s boxed trait type.

While C++ templates can generate very different code for different substitutions, in the end I’d expect for most of them instantiating them once for such dynamic type that would handle the rest with dynamic dispatch should be possible. It would not be as efficient as specific instantiations, but it would remain generic without requiring to compile more and more C++ code.

Also doing similar dynamic monomorphisation of Rust interfaces seems like a possible way to bind them to other languages that don’t have generics (like python, perl, ruby, java etc.), which would allow Rust to act as kind of common middle layer for combining with multiple other languages, which I think might be very useful.

[vadimcn]

It’s worth keeping in mind that the plugin is only infectious if you fail to encapsulate the C++ library, as well. Hard to say what will be the common case.

Yes, and I’d love to hear whether the community thinks this problem is worth tackling. For now I am assuming it is.

I’m curious to dig into this plugin a bit more. This is a good example of a plugin that goes beyond syntax extensions in terms of its requirements. What does it need precisely from rustc?

My prototype is not entirely implementable with plugins, sadly. It still required a change of the language to allow foreign functions to be generic. The other part is a post-trans plugin that enumerates all generated monomorphizations of foreign generics and produces the corresponding C++ instantiations for them.

As far as supporting templates go, I don’t see any way around the “infectious plugin” problem, except of course baking it into rustc (the ultimate infectious plugin).

Unfortunately, that is not enough. Even if you embed the C++ compiler inself into rustc, in order to generate additional instantiations, you’ll need [at least the the headers of] the original C++ library. I am afraid the latter requirement is not a reasonable one.

My current thinking is that the following compromise might be acceptable:

• The “wrapper” crate (i.e. the original Rust library wrapping a C++ library) should be able to pre-generate a number of C++ template instantiations as the author sees fit.
• Downstream crates can use this crate without incurring any extra dependencies as long as they stay within this pre-generated subset of types. Preferably, this should be enforced by the type system, for example via trait implementations for the allowed combinations of types (as I tried to illustrate here).
• Downstream crates may request new instantiations to be generated, but in this case they opt in to being dependent on the plugin, the C++ compiler and the C++ library.

Example: let’s say you set out to create a wrapper around std::map;
So you go on to define a generic Rust wrapper type, CppMap<K,V>, and use the cpp! macro in implementations of its methods to call out to std::map.

You also specify that the underlying template should be pre-instantiated for (K,V) in (i32,i32), (String, String), (i32, String) … and so on.
Downstream crates may freely use CppMap with those combinations of type parameters. To make sure of that, there is a trait CppMapParams, which is implemented only for those type combinations, and CppMap is constrained on implementations of this trait.

Later, somebody else creates a new type Foo in their crate, and really would like to use it as a key in CppMap. So they reference the “cpp” plugin from their crate, then write something like cpp_impl!( CppMap for (Foo,i32), (Foo, String), ...), which kicks off generation of the additional std::map instantiations. It also implements CppMapParams for (Foo,i32), (Foo,String), etc.

[nikomatsakis]

Unfortunately, that is not enough. Even if you embed the C++ compiler inself into rustc, in order to generate additional instantiations, you’ll need [at least the the headers of] the original C++ library. I am afraid the latter requirement is not a reasonable one.

Why is that unreasonable? That seems pretty typical to me, actually, of working with C++ templates. But also, since we currently build all Rust code from source, if you have the headers available to start, you’ll have them available the whole way through.

[vadimcn]

Why is that unreasonable? That seems pretty typical to me, actually, of working with C++ templates. But also, since we currently build all Rust code from source, if you have the headers available to start, you’ll have them available the whole way through.

You are right, if everything is being built from source, this is not a problem. But at least on Windows, distribution of pre-compiled libraries is not uncommon, and this is the scenario I was worried about.
If this isn’t something that Rust aims to support, the late template instantiation option is the way to go, no doubt.

[nikomatsakis]

But at least on Windows, distribution of pre-compiled libraries is not uncommon, and this is the scenario I was worried about.

I assume that these pre-compiled libraries must contain the header files in some form, at least if the libraries expose templates that can be instantiated in an open-ended way (versus to some fixed set of types). Basically, C++ has the same problem we do here, since they also monomorphize.

As far as Rust goes, we certainly aim to stabilize the ABI (eventually), and that should make distribution of binaries possible. However, there are a lot of advantages to building from source, so my expectation is that this will remain the primary way of distributing libraries. Basically there are a lot of modes one might use when compiling, and if you use binaries, you have to make these choices on behalf of your consumer: e.g., what target? debug info or not? what allocator? can we do LTO? do you want landing pads? overflow checks? etc. That said, to have the benefits, you don’t necessarily have to distribute SOURCE, it might also be serialized MIR or something similar (which would be roughly analogous to java .class files).

[vadimcn]

I assume that these pre-compiled libraries must contain the header files in some form, at least if the libraries expose templates that can be instantiated in an open-ended way (versus to some fixed set of types). Basically, C++ has the same problem we do here, since they also monomorphize.

For C++ - yes. Rust, however, abstracts a lot of that away by storing AST for generics in crate metadata, so from the user’s point of view Rust crates are self-contained. With C++ in the mix though, this abstraction breaks.
So yes, Rust ergonomics wouldn’t be worse than that of a comparable C++ library, but IMO the bar could be slightly higher.

What are your thoughts regarding the generic foreign functions? Would something like this be considered for inclusion in the language for the sake of c++ template interoperability? (note that this feature is useless on its own, it depends on a codegen plugin)

[eefriedman]

What are your thoughts regarding the generic foreign functions? Would something like this be considered for inclusion in the language for the sake of c++ template interoperability? (note that this feature is useless on its own, it depends on a codegen plugin)

There isn’t anything fundamental about foreign functions which prevents us from allowing generic foreign functions. The problem is that the C ABI has no notion of generics. We don’t want to allow people to write extern "C" { fn f<T>(); } because we can’t translate it to anything useful.

[comex]

The biggest issue with this seems to be the “infectiousness” of generic FFI: if a generic function, that internally depends on a generic FFI API is exported from the current crate, downstream crates will be able to cause new monomorphizations of the FFI stub to be created, and thus will become dependent on the codegen plugin, the C++ compiler, as well as the C++ library all being around to generate code for the new monomorphizations.

Is this really a big problem, given Cargo? Everything is compiled from source, so, generic exports or not, anyone using the library has already downloaded and run the compiler plugin; requiring client crates to also use it creates no additional logistic issues (as opposed to general usability issues, which depend on the hypothetical new plugin architecture). The plugin, for its part, can depend on a crate containing a copy of Clang, which can depend on something like the current ‘gcc’ crate to invoke the system compiler to compile Clang itself - likely to be somewhat annoying to deal with in practice due to compilation time and binary size, but doable. As for the presence of the C++ library, the situation doesn’t seem much different from C libraries today - either you bundle the whole library into a crate, or you look for it already installed somewhere on the system, in which case the headers will be there anyway if necessary (because C++ users need them too). The latter route would risk the annoyance of forcing everyone downstream to manually install the C++ library, but this is exactly the same today for a C library.

Edit: I really don’t think distributing binary Rust libraries is something worth optimizing for.

[nikomatsakis]

I’ve heard through the grapevine that the C++ committee may be making an effort to define a “portable subset” of C++ that can be used across C++ compilers etc. This may be a useful starting point for us, or at least an interesting data point.

[bfops]

I’m a hobbyist gamedev, and I’ve been using Rust for ~18 months on a near-daily basis. For me personally, having some sane story for C++ interop is a huge deal. It’s a pretty big reason that I’ve abandoned my main Rust project. Lots of wheels are being reinvented in Rust, and given enough time, maybe the need for C++ libraries could almost vanish, but that’s not compelling for me. Right now, the space of libraries-that-are-implemented-better-in-C++ is huge, and it was a lot of work to figure out which wheels had been reinvented in Rust, how well, how recently, how fully, and whether I should take a stab on my own.

tl;dr I still plan to use Rust for small projects and more systemsy things, but I would actively recommend away from it for serious gamedev, and that makes me sad.

[Leo2807]

I think we’re dealing with 2 different problems here:

1. Making Rust code available to C++.

2. Calling C++ from Rust. We can this in two subproblems:

   1. Representing classes/other arbitrary types in Rust

   2. Mapping these representations to calls to C++ code

For #1 there’s a workaround by making Rust available as standard C symbols using the known ffi methods.

I think #2.1 is a language-independent problem, so it might be better to implement it as one. I came up with two solutions:

1. Allow compiler plugins to define new types and have them handle the compilation of these types.

2. Make wrappers around foreign code (either manual or automatic)

My personal favourite is #1 because it allows for direct use of a library.

An example of what it could look like:

#![feature(plugin)]
#![plugin(cxx_interop)]

fn main() {
    include_cxx!("std::vector");

    let vec = cxx::std::vector::new<isize>();
    vec.insert(vec, 200);
}

But that would probably be very difficult to do. So #2 still has a chance.

[Riateche]

I’m aiming to create Rust bindings for Qt library. I’m currently developing a binding generator. There is still much work to be done, but some basic things already work and it looks promising. I’m working on it at https://github.com/rust-qt/cpp_to_rust.

[vadimcn]

I am not familiar with Qt API. Does it use macros, overloading, inheritance and templates? If so, I’d be curious to know, how are you planning to map these things into Rust?

[Riateche]

It doesn’t use macros much. Most of them are just used for shortening the code. There are some features based on macros and Qt’s own C++ preprocessor (moc), and they’ll require special treatment. I think it will be possible to support everything important, though.

Qt uses overloading a lot. Overloading is emulated with traits implemented over tuples of arguments. The only drawback is that instead of writing method() and method(arg1, arg2) the user has to write method(()) and method((arg1, arg2)). Rust calls the FFI function corresponding to supplied methods automatically. This system is already implemented. I also intend to add alternative overloading method based on Option type when possible.

Inheritance is also used a lot in Qt. Supporting C++ inheritance includes 3 things:

• Functions for downcasting and upcasting for available pointer types. Not implemented yet, but easy.
• Methods inherited from base classes are added directly to wrappers of derived classes. It’s kind of a hack, but it provides easy access to inherited methods. It could be replaced with trait-based method injection, but I don’t like it because the user would need to bring the trait into scope.
• Deriving from C++ classes is the most difficult, but also possible. The user will have to define extern “C” functions with specific signature and pass them to a special class constructor. These functions will be called as if they were reimplemented virtual methods of the class. The functions will receive pointers providing access to protected methods of the class. Maybe this will be made more convenient using Rust macros.

There are also issues related to automatic type conversion in C++. Functions and methods may receive pointers to base classes, and pointers to derived class will be automatically converted, but this is only one case. Qt uses automatic conversion between integers and enums, for example. Rust does not do any of this. It may be emulated with traits (like AsRef), but I’m not sure if it should be done.

Templates are my current task. Qt doesn’t use templates much. They are mostly used for vectors, smart pointers and so on. Instantiation of custom types from Rust side is currently out of the question. I want to generate wrappers for any template instantiation that can be encountered in API, in order to provide access to objects the user may acquire. I also want to organize inter-library template instantiation because Qt is separated to multiple libraries and they use templates of the QtCore library.

The bottom line is, Rust is good enough to emulate almost everything an API may need. It often is very inconvenient to do, but I’m writing an automatic code generator, so I can do things that are very inconvenient to implement but easy to use. Also, Qt’s API is really huge, and manual implementation is not an option anyway.

[hayd]

It might be worth looking at Cxx.jl which provides C++ interop for julia.

[emoon]

Funny I just started this over here https://users.rust-lang.org/t/pre-implementation-feedback-for-qt-with-rust/7300

I’m willing to help out with this.

[system]

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.