Solve `std::os::raw::c_void`


#21

It’s great to have this discussion, even though I think I have a different opinion on how to handle public dependencies. Rather, if they are to be handled so strictly, I’d rather not have any public dependencies at all.

We need a good solution for the c_void issue before libc 1.0 is released, it’s lucky at least that we caught it before then. I would prefer a solution similar to libcore. Maybe a new crate, something that isn’t all of libc.


#22

Like Python’s ctypes. That’s even a decent name, and someone already started a crate, but the repository is now 404.


#23

Yeah, I think this is a topic that needs a lot of attention (it’s a major pain point in the ecosystem today), and the core team (particularly @wycats) hopes to get a separate conversation going on this topic in the near future. Stay tuned!

Agreed. After digging in more deeply and re-reading this thread, here are my thoughts:

  • There are essentially two reasons for types like this to show up in std:
    • For “lowering” std types to their underlying C types – particularly used in IO. Types like c_void are already used this way.
      • These lowerings are essential, and mean that we’re committed to shipping many C types in std
    • For making FFI “first class” – so that the out of the box experience with Rust includes some basic FFI tools.
      • Reasonable people can argue whether these types belong in std, but many are stable already, and it’s long been the libs team vision to provide these kinds of basic tools in std.
  • The above argues to me that at least some of std::os::raw will always need to exist, and we should therefore not attempt to deprecate the module.
  • Note: there is a distinction between basic types (or other general “tools”) for FFI support, versus bindings. I personally imagine std's FFI support falling firmly into the former camp.
  • If we don’t deprecate, how do we solve the compatibility problem with libc? The essential issue is that libc should not require std.
    • We could stabilize libc as a dependency of std, but that’s a very heavy hammer! It would also be difficult to provide a smooth rollout.
    • We could move the bindings into core, as suggested on this thread.
      • Some feel, however, that core should be restricted to the, well, most core aspects of pure Rust, and that therefore FFI concerns belong elsewhere.
    • We could provide the bindings through an additional ctypes crate that sits between core and std.
      • libc and std would both depend on the crate, and reexport the c_void definition.
      • We have plans for stabilizing other such crates anyway.
      • It should be possible to make this move with zero breakage to the ecosystem – just suddenly more types are compatible.
        • We would need a feature flag for libc, or else Rust version detection.

I personally see very little downside to introducing a ctypes crate as part of the std facade. It’d be quite small, at least initially, but there may well be scope to grow our story here. This also gives a logical place for people to get basic C FFI definitions without using either std or libc (as may be appropriate for very low-level code).

Thoughts?


#24

You can do this with build scripts (example: scan-rulesbuild.rs).


#25

There are also some people who are interested, for various reasons, in Rust programs without libc at all. I’m fairly sympathetic to this, and it would be a shame if we locked ourselves out due to exposing libc stuff in a way that always requires a libc.


#26

Not if you link against APIs that only use explicitly-sized integers and size_t.

Usable definitions for void and char are important and general enough to be a part of libcore, as these form the base of the memory model. C89 integers (short, int, long) are less important as they are basically a language concern.


#27

If the ctypes don’t go into libcore, then libctypes sounds good.

There’s one appearance of c_int in libcore since it uses memcmp (but declares it with a return type of i32 so far). Memcmp is not quite an llvm intrinsic, yet the optimization passes recognize it, so it’s a relatively special cased libc function. Rustc or llvm could fix it by providing it as an intrinsic, then we don’t need to scramble for a c_int in libcore.


#28

I’m curious if anyone can elaborate further on this argument. There have been lots of statements that these types “don’t belong” in libcore, but I’d like to better understand the rationale. @alexcrichton or @brson?


#29

memcmp is sufficiently close to an intrinsic that using the actual return type instead of a type alias is not that much of a problem.


#30

I am not sure this is the best formulation of this argument, but I’ll give one of them a shot.

If libc stuff is in core, then Rust’s future is forever tied to C. In other words, look at something like Redox: if we’re building a new Rust universe of systems stuff, by adding libc in, we are still eventually perpetuating those interfaces. Of course, this doesn’t mean that we should never work with C stuff, but having it in an external package means that those who wish to go all-in on pure Rust don’t need to also build a new libc in it as well.


#31

c_void is more of an LLVM thing than a C thing. I think we should have memcmp be an intrinsic (maybe renamed as compare_memory or something).


#32

Defining fundamental C types in libcore need not require libc at all. It’s just about platform ABI, and in theory things like Redox could even make their own decisions in the target spec for how to host C programs, e.g. LP64 or LLP64.


#33

I think you’re misunderstanding the suggestion. Clearly libc itself can’t be part of core. The proposal is just plopping the contents of std::os::raw into core – just a few type definitions. IIRC, some of the Redox devs have explicitly asked for this, because it can facilitate FFI to C.

The pushback I’ve heard is more along the lines that this kind of FFI support shouldn’t be part of core, because core should only talk about Rust (or something along those lines).


#34

I think in the end it doesn’t matter much whether those type definitions are in core or in a separate ctypes crate. The only practical difference would be an extra line in Cargo.toml.


#35

Agreed. That said, it also seems simpler for everyone if it just lives in core, which is why I’m trying to understand the argument against that.


#36

The difference is that if these types go to an external crate ctypes, and ctypes ever gets a breaking change, then the whole ecosystem will need a major version bump if it wants to update its version of ctypes.


#37

I’m afraid that I don’t have a killer and concrete argument against adding the types to libcore, but my gut feeling is that this is not the right step to solve this problem. Some somewhat scattered thoughts I have on this are:

  • As I mentioned earlier, you can pass flags to C compilers to change the meaning of these types, a concrete case being you can change the signededness of char in C. That means that any definition we write down in libcore will inevitably be wrong. I was hoping to handle this by perhaps adding feature flags to libc on crates.io in the future.
  • Types like c_long are typically matching the pointer width on Unix platforms, but on Windows it’s 32 bits. That’s just plain weird, and I personally would feel pretty bad about persisting this weirdness for all of time in libcore. We as the authors libcore would probably never make a decision like this in a void, but otherwise we have to say that “the long type” is 32-bits on windows, but matches the pointer width everywhere else.
  • core is the theoretical foundation for all Rust code in existence. The question of whether core is usable and robust is always yes, and if not there’s a bug that we need to fix. I’m afraid that the addition of these types in libcore may perhaps one day lead us to a bug report that we just can’t fix, so all of a sudden libcore isn’t usable for “literally every purpose”. This fear is unsubstantiated and vague, however, of course!

I think I’m not comfortable yet on adding these types to libcore, and I’ll try to think on it some more to see if I can articulate something better.


In terms of moving forward, I would prefer one of two strategies

  1. Deprecate the types in libstd. I think there’s at least an avenue to do this, although it seems there’s enough pushback to make this option moot.
  2. Stabilize shipping libc as part of the main distribution. This has its own complications, but is basically becoming true to what the dependency structure actually looks like and would solve the problem as well. We could one day in the future add a ctypes crate under the libc crate, but I don’t think we need to do that at this time.

#38

What problems will using the wrong signedness for char put you in? Both types have the same size and alignment, and char is out of the strict aliasing game. If LLVM has a concept of char, then libcore needs to follow that concept.

I would prefer not to expose the C flexibly-sized integer types for exactly this reason - just c_void and c_char. Every “cross-language” API should be using the explicit-sized integers anyway.

In the worst case, we will have types in libcore that don’t match up with the types the local C compiler uses. Big Deal. It’s not like using liblibc would help you.


#39

That’s exactly LP64 vs. LLP64. From Rust’s perspective, I’d just avoid saying anything about long matching pointer width, and only say that it matches the platform’s C ABI.


#40

ABI-wise you don’t have a problem, but if you define a min function in C and a min function in Rust you’ll get different results obviously, which means that we’ll inevitable receive a bug report saying that something needs to get fixed.

It’s true that the failure mode isn’t necessarily catastrophic, but it would perhaps end up with a meme that libcore C types are wrong and shouldn’t be used, in which case why did we add them in the first place?

True! It seems weird to declare this in libcore, however, to me. There’s no real reason to decide one way or the other in our perspective?