Need custom calling convention for COM

[retep998]

After someone brought this to my attention, and I did a bit of digging, it turns out that COM is almost but not quite identical to stdcall. The difference is that the C++ COM methods will always return a struct via a pointer, whereas stdcall, if the struct has only two pointer sized (or smaller) members, it will return it by value in eax and edx for 32-bit, or by value in rax for 64-bit. Although this is used very rarely, it is a huge problem when a COM function does return a struct, such as ID3D12Device::GetAdapterLuid.

Therefore I propose creating a new custom calling convention, based on stdcall, except it always returns aggregates by pointer.

[Diggsey]

This doesn’t seem to be related to COM at all - it affects all stdcall functions returning a small aggregate. AFAICT based on what little information I could find, the MS stdcall ABI differs between C and C++ when returning aggregates (suggesting MSVC itself will generate different code for the two languages!).

Since COM interfaces are more commonly implemented in C++, they usually use the pointer-passing version of the calling convention.

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=64384

https://www.winehq.org/pipermail/wine-patches/2014-September/134351.html

[retep998]

It only applies to C++ methods that are stdcall. If I create a normal stdcall function in MSVC C++ then it returns small aggregates by value. So it isn’t a difference between C and C++, but rather between functions and methods.

[retep998]

Anyway, it seems LLVM does have the support we need, but Rust has to be modified to emit the proper LLVM IR for LLVM to generate the code correctly.

https://gist.github.com/retep998/604861ea06aa984ee6c7

Any proposals on how the syntax for signifying this should look? Honestly I don’t really care too much what the syntax is, I just want something that works without having to manually hack up the function signatures.

[mzabaluev]

How do you represent a C++ method in FFI? Does name mangling have to be done explicitly on the Rust side?

If you just fake that as a C function with the first argument exposing this, that may run afoul of some C++ ABIs which actually make difference for methods. As is in evidence above.

I vaguely remember that COM tools included a generator that could produce plain C bindings out of IDL or some other form of introspection (typelib?). Maybe that should be the preferred way of binding COM APIs?

[mzabaluev]

This might be helpful:

http://stackoverflow.com/a/4468584/2217151

A 5-part tutorial from 2006, looks hellish: http://www.codeproject.com/Articles/14183/COM-in-plain-C-Part

[retep998]

Name mangling isn’t an issue since COM is vtables. There’s no symbols to link to.

Using the C bindings produced from IDL won’t work. They actually suffer from the same issue. Believe me, we tested this and using MSVC with the C bindings for those certain functions caused incorrect behavior.

[retep998]

Those tutorials aren’t very helpful since I already have COM working fine. It is only some rare functions in COM that return an aggregate value that cause incorrect behavior. Neither of those links cover that case, and I need some way to tell Rust that it isn’t just an stdcall function but a method and thus modify the LLVM IR it outputs so that case can be handled correctly.

[mzabaluev]

Name mangling isn’t an issue since COM is vtables. There’s no symbols to link to.

Thanks for helping me remember. So you represent the vtables as structs of extern "stdcall" fns matching the method signatures, with addition of this as an explicit first parameter? This may not be entirely correct use of "stdcall" as a C ABI. Indeed, it looks like a new calling convention label is needed. Or, maybe, a C++ repr could be introduced to apply to a thin trait:

Blog post: Extended Enums and Thin Traits

I wrote a blog post that compares the "Extended Enums" stuff I was describing with the "Thin Traits" (my term) stuff that @aturon was describing: http://smallcultfollowing.com/babysteps/blog/2015/10/08/virtual-structs-part-4-extended-enums-and-thin-traits/

[retep998]

A struct of extern "stdcall" fn with explicit this parameters is how it is represented in the C bindings provided by Microsoft, which works fine for 99% of COM but fails for methods that return structs. Notice this does mean that in C using the official C bindings for those methods is actually broken. At least in Rust we can try to be a bit better than C and have a proper calling convention for this.

I’ve opened an issue in the rfcs repo: https://github.com/rust-lang/rfcs/issues/1342

[mzabaluev]

Might the calling convention discussed here be __thiscall?

How do the C bindings (vtable structures in the non-__cplusplus API sections) generated by MIDL actually look like? Sorry for asking all this, but I don’t have the time to go down that rabbit hole myself.

[retep998]

Using __thiscall in the C bindings doesn’t fix the issue. Rust doesn’t expose "thiscall" at all. Regardless, the C++ COM interfaces are explicitly declared using __stdcall so they’re not using __thiscall.

If you want to know what the C bindings look like, here’s an example.

https://gist.github.com/retep998/cf0246b7b56723fdc149

STDMETHODCALLTYPE is just defined to __stdcall.

[mzabaluev]

STDMETHODCALLTYPE is just defined to __stdcall

Are you certain that there is no alternative definition for non-__cplusplus compilation?

[retep998]

Are you certain that there is no alternative definition for non-__cplusplus compilation?

Yes I am certain that there is no alternative definition.

[nikomatsakis]

So, in an IRC discussion that @retep998 forwarded to me (which I’ve misplaced the link to), it was mentioned that you would wind up needing a “method” variant on every calling convention. This seems like a drag, for sure, though I wonder how true it is – iow, is this strategy of modeling an object as a struct full of fn ptrs something widely used in Windows land, or is it specific to COM, and if so, do the vast majority of APIs use stdcall?

[retep998]

Using C++ from C is a very difficult thing that people rarely do. However, using COM from C and many other languages too (I’m looking at you Delphi) is a rather common thing. COM is designed to provide a stable ABI that can be used across languages. I have never seen any COM interfaces use something other than stdcall. Having a struct full of fn ptrs is basically what COM is and I’ve never seen that used for anything other than COM. I think it would be a reasonable tradeoff to only support the method version of stdcall now and ignore the various other calling conventions, and let a proper C++ integration proposal deal with those.

[system]

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.