Because size_of::<bool>() is 1, reference need to be able to replace the whole byte.
One way that packing like you describe could be legal would be something like "copy-only fields" (A #[repr] to allow an enum containing enums to use a single discriminant - #5 by scottmcm) so there's always the opportunity to run code, and thus it's work fairly easily.