The flat attribute seems like it might be confusing in the case where someone wants an actual bitfield. I suggest the two features stay separate.
I already strongly disagree with this assertion. Bitfields in C are a pain to handle, because they contribute even more issues to the already nontrivial memory model of the language. Bitfields are an incredibly easy way of introducing undefned behavior. Furthermore, most uses of bitfields that I have seen in the wild basically amounted to premature optimization.
Bitfields do not compose either, and they break the regularity of the type system. Bitfield "types" are not really types, because they cannot be instantiated, addressed, or read/written individually, only inside the context of the struct they are defined in. Consequently, supporting them would be a massive churn on the compiler and likely a source of many bugs. Considering all those factors, they are not really adequate for Rust as they contrast quite sharply with the existing design of the language.
4 Likes
Even if I agreed with that sentiment, bitfields would still be a necessary addition to Rust in one way or another: if nothing else, to be able to interoperate with C code which uses bitfields as part of its public ABI. It’s not too hard to find that in the wild.
Perhaps it might be more productive to list some specific difficulties we are likely to run into with this feature (so that we might perhaps avoid them during the design stage, before it’s too late), instead of responding to the proposal with blanket opposition. In fact, I kind of feel my draft’s ‘drawbacks’ section is rather under-developed right now, so I would certainly like to discuss those. Myself, I only have vague recollections of some uncertainties with regard to volatile, but I don’t think those apply here.
4 Likes
I listed some of those in the second paragraph. But if you are looking for even more, here are some which can be derived from the ones I mentioned:
- How do you handle bitfields in derive macros, given that several automatic impls need to borrow fields, and changing that would not only be udnesirable, but also breaking? Actually, the same is true for convenient manual impls. The most trivial example is
Debug, which is most easily implemented using theFormatter::debug_struct()function. This returns a builder that takes each field as a&dyn Debug. This interface would then be impossible to use on bitfield-containing structs. - In general, a completely new type of field requires all proc-macro authors to think of and correctly implement yet another case. Furthermore, code that aims to represent the type system of Rust in some way or another would need to deal with bitfield non-"types", the most notable concrete example being Serde.
1 Like
Is it not an option to simply fail to compile derive macros on structs with bit fields? If their use is niche, then people can take on the burden of manually implementing the functionality for just those structs. Derive should continue to work fine for types containing of such structs which would simply delegate to their children's implementations.
The exact same concern applies to #[repr(packed)], as you can't borrow those fields. Either the derive supports #[repr(packed)] structures, and has a path to copy to a stack temporary and can apply that same path for individually packed fields, or it doesn't, and it will fail to compile.
Bit-level or otherwise compacted fields are restrictive, yes. But unlike C, which just leaves you on your own to obey the restrictions, we have the advantage of a compiler that can enforce them.
11 Likes
I think at the very least, #[derive(Debug)] must be made to work. As a temporary workaround, I’d suggest making it perform the copy-to-temporary-storage trick for Copy types, and output a generic placeholder otherwise. This will need to be addressed more generally in the long term.
I'm not sure I understand why derive- Debug support is necessary; it's straightforward to implement by hand. We should be able to get away with supporting just the core marker traits ( Copy and auto-traits) and also Clone only if Copy .
I agree with all your arguments and yet, I don't reach the same conclusion. Despite all the quirks of bitfields, that make them fundamentally different from byte-aligned fields, I consider them a necessary evil, because Rust is a systems programming language and there are at least 2 cases where bitfields are mandatory and 1 case where I consider them to be unavoidable.
The mandatory cases are interacting with hardware, which may require bitfields and the other case is, stuffing different data in a single atomic object, because the hardware doesn't offer arbitrary-sized atomic objects and without exploiting the available space, atomicity cannot be achieved. The unavoidable case is serialization, if you work with very large data sets (10+ GiB, 100+ GiB or even 1+ TiB) and using a general purpose compression algorithm is not an option, anymore.
1 Like
Ironically, I think interfacing with hardware makes one of the weakest cases for bitfields. Such code can always be written with manual bit shifting an masking; since you need to exercise tight control over the memory layout of your data and often when memory accesses happen at all, and all code is platform-specific anyway, there isn’t that much to be gained by having it supported by the language in a generic way, especially without a stable ABI to go with it.
The derive is half the point of Debug. (The other half is a uniform interface.) Even in the minimal viable version of this feature, usable only for Copy enums without payloads (like bool), people would still need printing out debugging information to be as expedient as possible. Right now it’s a two-liner: one line for the derive, the other for dbg!. Bit packing should not change that.
1 Like
I think C bitfields should be left for bindgen to worry about. There's nothing special that compiler needs to do about them, and I don't think it's necessary to have first-class syntax for such an awkward feature. In the end, bitfield access is reading/writing larger integers, so it's fine to build an interface around that.
3 Likes
I review a lot of very bad C/C++ written by firmware people. I know what you're talking about. =)
My expectation of "hey this feature from C is great let's think really hard about it" should not be taken as "lol let's adopt it exactly as is"; my expectation is that we think really hard about what's useful about it and how to staple it to Rust without pulling in any of the historical stupid.
This is missing the forest for the trees a little bit.
(The only reason bitfields come up here because this is a convenient mechanism for packing things at the level of bits; in a world where we have a PackedArray type like I described above, there is an obvious path to bitfields-as-flattened-fields. Let's figure out flattened fields first, though.)
5 Likes
I managed to tinker this abomination:
tl;dr Zero-sized fields + const generics + repr(transparent) = this shit seems to work. Someone has to check if this is actually sound, but making a proc-macro for this seems trivial.
One downside is that implementing Copy is illegal for such bitfields.
UPD: Miri does not like it. Would using an array of UnsafeCell instead of u8 help?
I also tried something similar but:
#[repr(transparent)]
struct Foo {
a: (),
b: u8,
}
let foo: Foo = Foo {
a: (),
b: 11,
};
let a: &() = &foo.a;
// WARNING UB
let b: &u8 = &*(a as *const () as *const u8);
If the fields are rvalues, so that taking references to them automatically creates a temporary on the stack, most derives should "just work". Are there any built-in derives that wouldn't?
Considering all those factors, they are not really adequate for Rust as they contrast quite sharply with the existing design of the language.
If it were just bitfields, I'd say that, while it's important for a systems language to have some way to pack data into bits, Rust's macro system provides a decent existing alternative. (And personally, rather than a built-in bitfields feature, I'd prefer to have a way to overload property access, to allow for custom packing schemes without an ergonomics penalty compared to normal fields. Your taste may vary.)
But it's not just bitfields. I'd say that, when you look at examples like the OP's:
struct Foo {
a: Option<u32>,
b: Option<u32>,
c: Option<u32>,
}
...Option is fundamentally not acting as a zero-cost abstraction compared to what you would write by hand. Since this problem can occur as a result of generic instantiations, and also involves the interaction of multiple fields of a struct, it's not something that could be easily solved outside the compiler, even if we had more sophisticated metaprogramming functionality. But it can be solved inside the compiler, I'd argue, surprisingly easily, at least in terms of impact to the rest of the language. The fact that the same approach can solve a longstanding problem with packed fields is a nice bonus.
7 Likes
It feels to me that Packable should be a trait for which it is trivial to find implementations. Then to write
struct Foo {
a: impl Packable<Option<u32>>,
b: impl Packable<Option<u32>>,
c: Others,
}
Making explicit that the compiler is making choices when defining the type Foo.
Great, then I want to bring up a tangential issue related to bitfields: endianness. I.e. the order of individual bytes that make up numeric values in memory. Whenever I was busy with bit-fu, endianness was often a concern.
In practice, few people worry about big vs little endian anymore. All modern x86 and ARM chips ship little endian. Today the average developer will only ever run into endianness issues when writing low level netcode. This is why the byteorder crate includes a NetworkEndian alias, and I must tip my hat to them for catching that. 
Go back a few years though, and big vs little endian was a concern, at least for game console developers. The IBM and MIPS CPUs found in devices prior to the x86 and ARM console generations were big endian, where as PC's were little (ignoring PowerPC Macs).
You'd develop on your little endian PC, compile locally, then run builds on your big endian consoles. Data files were authored on the PC, and you had to be careful not to mix up your 16bit and 32bit values when sharing data files across platforms. It's true that it's not that expensive to byte swap, but consoles were almost always slower then their PC counterparts (at least for I/O), so preparing files for the target device was always a good idea.
Types containing bitfields were often used in conjunction with endian byte swapping macros (and unions). Defining an ideal data structure, headers and other packed data, then wrapping all bulk IO code with macros. Let's say you were making a chunked format with 16 bit section headers: 2 bits for type, and 14 bits for size in bytes: 16384 bytes for size would be enough for a 128x128 byte grid. Or if we know sizes will always be even, we can effectively describe double that in 14 bits. Anyways, my point is we could say a lot with bits, and bitfields resulted in more readable code than ANDing, ORing, and shifting constant values (macros) together.
Older graphics hardware benefitted from bitfields. On something like a GameBoy, there were fixed addresses on the bus you could directly write to to control what was seen on screen. Something called a "scroll register" was a value you could write to that controlled the x or y offset of a "character map" (I.e. a background). A value of 1 or 2 would offset it by 1 or 2 pixels. We'd create the effect of vertical or horizontal game scrolling by changing the value of this "register" every screen refresh. When the visible screen size was less than 256x256 pixels wide, 8bit scroll registers were enough. That said you needed 9-bit scroll registers for your "Advanced" consoles. The chip designers weren't going to waste valuable address space, so that 9th bit was often packed together with other flags and features, such as a bit for the character map width (32 or 64 wide), what base address to pull characters from, among other things.
It's been a while, but I'm remembering that you couldn't always trust bits to be safe to read first then write. I.e. in the 9bit example above, a spare bit might be used to start a timer when it's set to 1. When read, it returns a status of 1 meaning it's activated, but if you were to write 1 the timer would reset. Admittedly these were rare, and were usually nuances of networking not graphics, but the awkward behavior still stands.
Anyway I wanted to say I did really like the simplicity of C's bitfields. The syntax might not work for Rust, but I liked that could do something like this to describe a complex 32bit chunked header:
struct ChunkedHeader {
u32 type: 4; // 0-15 or an enum
bool flag1: 1;
bool flag2: 1;
s32 normal: 2; // +1, 0, -1
u32 size: 24; // 24bit number, enough room for millions
}
Things got especially nice with anonymous types and unions.
union ChunkedHeader {
struct {
u32 type: 4; // 0-15 or an enum
bool flag1: 1;
bool flag2: 1;
s32 normal: 2; // +1, 0, -1
u32 size: 24; // 24bit number
}
u32 as_u32;
u8 as_u8[4];
}
But we still needed our macros when reading/writing.
1 Like
I came up with two sound (miri doesn't complain) ways of kind of simulating bit fields
Using Deref
https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=3b23e373ef506318f248b4c016138275
- only allows two bit fields easily
Using an union
https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=50d439cba1d0f49e98984d1b220a9cd7
- probably the best way of simulating bit fields
- no way of telling rust "all fields of this union are always safe to access" so you need an unsafe block to access the fields
I think that Microsoft is looking at ways for the Rust compiler to not need unsafe in situations like that.
I do rather like how easy the union method is
I'm positive that assigning one field a value and then reading from another in a union is UB, at least in C.