Solve `std::os::raw::c_void`

[Amanieu]

I think in the end it doesn’t matter much whether those type definitions are in core or in a separate ctypes crate. The only practical difference would be an extra line in Cargo.toml.

[aturon]

I think in the end it doesn’t matter much whether those type definitions are in core or in a separate ctypes crate. The only practical difference would be an extra line in Cargo.toml.

Agreed. That said, it also seems simpler for everyone if it just lives in core, which is why I’m trying to understand the argument against that.

[tomaka]

The difference is that if these types go to an external crate ctypes, and ctypes ever gets a breaking change, then the whole ecosystem will need a major version bump if it wants to update its version of ctypes.

[alexcrichton]

I’m afraid that I don’t have a killer and concrete argument against adding the types to libcore, but my gut feeling is that this is not the right step to solve this problem. Some somewhat scattered thoughts I have on this are:

• As I mentioned earlier, you can pass flags to C compilers to change the meaning of these types, a concrete case being you can change the signededness of char in C. That means that any definition we write down in libcore will inevitably be wrong. I was hoping to handle this by perhaps adding feature flags to libc on crates.io in the future.
• Types like c_long are typically matching the pointer width on Unix platforms, but on Windows it’s 32 bits. That’s just plain weird, and I personally would feel pretty bad about persisting this weirdness for all of time in libcore. We as the authors libcore would probably never make a decision like this in a void, but otherwise we have to say that “the long type” is 32-bits on windows, but matches the pointer width everywhere else.
• core is the theoretical foundation for all Rust code in existence. The question of whether core is usable and robust is always yes, and if not there’s a bug that we need to fix. I’m afraid that the addition of these types in libcore may perhaps one day lead us to a bug report that we just can’t fix, so all of a sudden libcore isn’t usable for “literally every purpose”. This fear is unsubstantiated and vague, however, of course!

I think I’m not comfortable yet on adding these types to libcore, and I’ll try to think on it some more to see if I can articulate something better.

---

In terms of moving forward, I would prefer one of two strategies

1. Deprecate the types in libstd. I think there’s at least an avenue to do this, although it seems there’s enough pushback to make this option moot.
2. Stabilize shipping libc as part of the main distribution. This has its own complications, but is basically becoming true to what the dependency structure actually looks like and would solve the problem as well. We could one day in the future add a ctypes crate under the libc crate, but I don’t think we need to do that at this time.

[arielb1]

As I mentioned earlier, you can pass flags to C compilers to change the meaning of these types, a concrete case being you can change the signededness of char in C. That means that any definition we write down in libcore will inevitably be wrong. I was hoping to handle this by perhaps adding feature flags to libc on crates.io in the future.

What problems will using the wrong signedness for char put you in? Both types have the same size and alignment, and char is out of the strict aliasing game. If LLVM has a concept of char, then libcore needs to follow that concept.

Types like c_long are typically matching the pointer width on Unix platforms, but on Windows it’s 32 bits. That’s just plain weird, and I personally would feel pretty bad about persisting this weirdness for all of time in libcore. We as the authors libcore would probably never make a decision like this in a void, but otherwise we have to say that “the long type” is 32-bits on windows, but matches the pointer width everywhere else.

I would prefer not to expose the C flexibly-sized integer types for exactly this reason - just c_void and c_char. Every “cross-language” API should be using the explicit-sized integers anyway.

core is the theoretical foundation for all Rust code in existence. The question of whether core is usable and robust is always yes, and if not there’s a bug that we need to fix. I’m afraid that the addition of these types in libcore may perhaps one day lead us to a bug report that we just can’t fix, so all of a sudden libcore isn’t usable for “literally every purpose”. This fear is unsubstantiated and vague, however, of course!

In the worst case, we will have types in libcore that don’t match up with the types the local C compiler uses. Big Deal. It’s not like using liblibc would help you.

[cuviper]

Types like c_long are typically matching the pointer width on Unix platforms, but on Windows it’s 32 bits. That’s just plain weird, and I personally would feel pretty bad about persisting this weirdness for all of time in libcore. We as the authors libcore would probably never make a decision like this in a void, but otherwise we have to say that “the long type” is 32-bits on windows, but matches the pointer width everywhere else.

That’s exactly LP64 vs. LLP64. From Rust’s perspective, I’d just avoid saying anything about long matching pointer width, and only say that it matches the platform’s C ABI.

[alexcrichton]

What problems will using the wrong signedness for char put you in?

ABI-wise you don’t have a problem, but if you define a min function in C and a min function in Rust you’ll get different results obviously, which means that we’ll inevitable receive a bug report saying that something needs to get fixed.

In the worst case, we will have types in libcore that don’t match up with the types the local C compiler uses. Big Deal.

It’s true that the failure mode isn’t necessarily catastrophic, but it would perhaps end up with a meme that libcore C types are wrong and shouldn’t be used, in which case why did we add them in the first place?

That’s exactly LP64 vs. LLP64. From Rust’s perspective, I’d just avoid saying anything about long matching pointer width, and only say that it matches the platform’s C ABI.

True! It seems weird to declare this in libcore, however, to me. There’s no real reason to decide one way or the other in our perspective?

[bluss]

I guess it’s a bit silly, but c_void could alone go into libcore.

[ahmedcharles]

I think c_void and c_char should be in core. The fact that compilers allow for changing the signedness of char via a commandline flag isn’t really an issue unique to Rust. The platform has to pick a default for compiled code and overriding that default will cause the same incompatibilities with existing C code as it would with existing Rust code.

As for the other types, I would initially be conservative and put them in a ctypes crate between core and std.

[aturon]

Many of the strategies we’ve been discussing recently involve putting the definition of c_void “under” std, either via core or a new ctypes crate. Then both std and libc could use this shared definition.

In the initial post, @bluss claimed that repointing libc to a shared definition like that would require a major version bump in the crate. (There was a further claim that a major bump will eventually be needed for one reason or another, but let’s put that aside for the moment.)

Questions:

• Is such a major version bump strictly necessary? Why?
• If we have to do a major bump to libc, isn’t that likely to cause at least as much pain as deprecating std::os::raw in favor of libc would?

[retep998]

In my opinion, changing libc::c_void to be a reexport of a c_void from a higher up crate would not be a breaking change. I can’t imagine how any code could rely on the assumption that the type isn’t a re-export and be broken by that change.

[bluss]

• My idea was that libc 0.2.16 and libc 0.2.17 need to have compatible c_void definitions. That seems to be a misunderstanding actually, since you wouldn’t use both of those in the same project (and if you do, their c_voids are incompatible anyway).
• A major bump in libc is planned anyway (1.0), so that’s not an issue.

[aturon]

Thanks @retep998 and @bluss!

So with that out of the way, I feel like @alexcrichton’s arguments against putting types in core don’t apply to c_void, which happens to be the only type for which the location matters.

Thus, if we want to solve this problem quickly and with minimal pain and commitment, the most plausible routes seem to be one of the following:

1. Introducing a ctypes crate under std with all of std::os::raw in it.
2. Adding c_void alone to core, which doesn’t require weighing in on various C issues but solves the main problem.

If we are motivated to remove std::os::raw from std (perhaps for the reasons @alexcrichton laid out), I think the ctypes approach is the best solution. We could take the following steps:

• Introduce the ctypes crate into the Rust distro, and make std re-export its definitions in std::os::raw.
• Update libc to also re-export those types, with a minor version bump.
  • Note: this is a bit tricky since it requires a particular version of the Rust distro to work; we’d probably need some way to deal with that, either via detection or feature flags.
• Now, the world of libc and std::os::raw types are fully compatible, solving the original problem.
• At this point, we can safely deprecate std::os::raw in favor of ctypes, and perhaps at some later date remove those deprecations in a major version of Rust.
  • This deprecation no longer causes massive pain, because you can switch to the libc (or ctypes) definitions without breakage.

Thoughts?

[alexcrichton]

Update libc to also re-export those types, with a minor version bump.

Overall that plan seems reasonable to me, but diving into the details on this one may show that it’s pretty tricky. We could actually switch these to reexports today, except that libc now has a feature to support #![no_std]. I would feel pretty bad about reexporting if you link to std and defining if you don’t, and it also seems like it could cause confusion/breakage.

That is, we may not be able to switch libc to reexport until the ctypes crate is stable, unless we’re willing to define some of the time and reexport the rest of the time.

[jethrogb]

That is, we may not be able to switch libc to reexport until the ctypes crate is stable, unless we’re willing to define some of the time and reexport the rest of the time.

Are there any arguments against doing this?

[alexcrichton]

I would personally avoid doing so as it seems “pretty weird” that sometimes libc defines these types and other times it doesn’t. There is indeed subtle breakage that can happen by defining a new type vs reexporting another. A silly and contrived example is impl LocalTrait for libc::c_void plus impl LocalTrait for std::os::raw::c_void, where that will compile with #![no_std] and it won’t compile if libc links to std.

That, and it still doesn’t solve the problem of “one source of truth” for these types (which is the original crux of this issue). If we do that then I would think that we’re basically where we are today, with the “pain” just shifted somewhere else.

[bluss]

I think it sounds good. A ctypes crate is more than c_void needs, but from the discussion here it sounds like the best solution.

[alexcrichton]

Hm so another possible snag is that ideally libc never links to libstd. That is, the use_std feature just shouldn’t exist because the library never needs std (it was just left in for backwards compatibility in niche use cases).

[jethrogb]

@aturon @alexcrichton What’s needed to move forward on this?

[jethrogb]

@aturon @alexcrichton another ping now you’re back from your conference