Why even unused data needs to be valid

This almost sounds like a loophole in the proposition that "safe code can never produce UB". It sounds like I can create a struct in safe code that violates the safety invariants of that type, even though the individual components are valid, which could eventually, somewhere in some code lead to UB without ever involving unsafe code. Or are you saying that only possible way to get UB here is that some unsafe code would have to operate on this struct that has invariants while making the assumption that the invariants are met, which violates the requirement that "unsafe code cannot rely on safe code to maintain invariants"? Somehow, that seems not quite right because "unsafe" code must rely upon the safe code within the privacy boundaries of the module to maintain invariants. Is that not the case?

EDIT 1:

If not in safe code what about if unsafe code creates a struct (or enum) value where the invariants of the struct are not correct. If I understand correctly, this isn't "Language Level UB" so unsafe code is not required to ensure the invariants are met, but, this can lead to UB elsewhere (even in safe code). Doesn't that mean that the requirement that composite invariants must be maintained at unsafe boundaries by unsafe code or it is UB, or, in this case is it only UB when the invalid composite value is used?

EDIT 2:

This gets me thinking about how "unsafe" is used and whether some additional requirements are warranted. Unsafe today should rely upon the privacy boundary of the containing module to ensure that no values that the unsafe code depends upon are modifiable directly, without going through a method/function defined within the module, which modifies those values in a way that maintains the necessary invariants for the unsafe code within the module. This means that "unsafe" code is relying upon "safe" (though private) code to maintain invariants. Should it be that structs must declare which fields are used in invariants of the unsafe code within the module and it should be prohibited to mutate those fields, even within the module, without using "unsafe" AND that "unsafe" code that uses/references values of structs that are not declared as part of the invariants would be a compile-time error? This would prevent "unsafe" code relying upon anything other than a specifically declared contract where upholding the contract always requires "unsafe" because a violation of that contract could lead to UB?