Regarding w65: I find compiling Rust for the SNES interesting as an idea but I see no reason that, even if Rust is a viable target for that platform, that it makes maximal use of the theoretical capabilities of the hardware. The Rust language is free to impose bounds that constrain code to obey its model, not the hardware's. e.g. Rust is far stricter about unaligned pointers than a lot of hardware, even though most of the "normal" hardware Rust was originally designed for can cope well with such a demand, and certainly doesn't, say, fault. This of course makes the logic simpler and makes the code portable to architectures which do fault, though as I understand, even most embedded architectures we support can even cope with some slightly overly vigorous accessing (it being on a more instruction-by-instruction basis).
This is really the case of any language higher-level than assembly. However, part of the SNES-Dev project is to see how much of an impact it has.
The reason I wanted rust as part of the project in the first place, was because I believed the project could benefit from it, and from analyzing the impact of the forced memory safety that comes along with it.
This is the case with the w65 abi anyways. It imposes an alignment requirement of primitives equal to size up to 2 bytes, then pointers at 4 bytes (though it may be necessary to move the maximum primitive alignment to 4 bytes). This was done to avoid issues where the same "scalar unit" which is accessed by the CPU is allocated accross a bank, or more recently a page (as the former has some fun semantics in a few cases involving wrapping, and the latter incurs a cycle penalty); the CPU itself cares nothing about alignment. This incurs some space penalities, which can result in time penalities as well, but it does come with advantages.
Yes, thatās why I was asking about implicit coercion, it doesnāt help much. The discussion about explicit coercion from pointer to usize and vice-versa is a different one. I initially thought a type uptr = usize on āregularā platforms would be a good option, too, but thereās problems. You canāt add trait implementations for both usize and uptr on such platforms anymore without using cfg everywhere, so it becomes harder to write platform-independent code.
āBreakingā existing code for new non-āregularā platforms if the code uses explicit pointer <-> usize coercions is honestly something that canāt be avoided with any approach. (But it isnāt really ābreakageā either, you just donāt get support for those new platforms.) But at least we can have a situation where you know that if your code compiles for a non-regular platform, then it will also compile for a regular platform.
With a separate new uptr type for most crates, this will just be yet-another new integer type. A high-level safe-code crate like e.g. num can add uptr support without any need for weird cfgs and call it a day.
I think this is inevitable. Even if some compiler magic was used to allow usize and uptr to support separate trait implementations, it would still lead to coherence problems because we want traits on usize to apply to uptr (on pain of backwards incompatibility when some function fn() -> usize is changed to fn() -> uptr and is now no longer eligible for some user trait impl Trait for usize that does not know about uptr).
Iāve said it twice in this thread and Iāll say it again: Iām not aware of any function in std that needs to be changed in this way, but please correct me if Iām wrong. Outside of std, I assume itās rare that such a change needs to happen in any public API; if you know of any crate that would need to make such a change in order to get rid of pointer<->usize coercions, please tell me.
If there is no (public) function that needs to be changed then then there is no āpain of backwards incompatibilityā either. If there is the rare / odd crate out there, thereās always the option of releasing a new major version, or indroducing a new fn() -> uptr version of the function with deprecation of the old one (together with a cfg removing the deprecated version on the new non-āregularā platforms).
Yeah, actually, thinking about it, it seems to be that the question is about what functions do internally, since no one really casts a function to usize and then exposes that to external API. Instead our APIs expose/consume "usize as in size_t". "usize is uintptr_t" is mostly an internal implementation detail.
libc does this. It has the definition type size_t = usize;... And then proceeds to use that to accept a fn-pointer parameter in signal (which is straight up undefined behaviour - function pointers are 100% not FFI compatible at all with usize)
What do you mean by "undefined behavior", exactly?
This is implementation-defined behavior, and it is well-defined for any implementation of Rust for a von Neumann platform where usize::BITS == size_t::BITS == uintptr_t::BITS (and, er, POSIX relies on it, as does Windows).
Program behavior is not defined by optimizations or by "what the compiler does". So I can't agree to the way you are framing this here.
But I agree that this "loss of information" can be mostly ignored by programmers (but not by compiler writers). That said, the emphasis is on mostly -- and there is an unexpected cost for this. This example shows that a ptr-to-int cast whose result is unused cannot be removed from the program in languages with "interesting" provenance like C's restrict, LLVM's noalias, or Rust's reference guarantees. And this example shows that transmuting a pointer to an integer is a bogus operation; a concern programmers can not just ignore.
We cannot remove support for int-to-ptr casts entirely from Rust, so we cannot entirely get rid of this cost. But I still think it is worthwhile to migrate as much code as is possible away from this cast: it will make teaching our memory model and pointer provenance a lot easier, and as this discussion shows it will also help support architectures like CHERI.
Note that the C-level API for signal and sigaction is strictly and accurately typed.Ā¹ I don't know what prevents libc from declaring
type sighandler_t = unsafe fn(c_int) -> ();
but whatever it is, I would like to think it's fixable without becoming entangled with the rest of this proposal.
The dlsym interface may be more troublesome in this context, since it requires arbitrary function pointer types to be convertible to C void * and back with no loss of information, but this requirement applies to void *, notsize_t.
Ā¹ except for the third argument to 3-argument handlers, but that's a case of a function taking an argument declared as void * instead of the concrete pointer-to-data type it should have had: much less problematic.
It also doesn't require compatibility of void* and fn(): just the fact they can be interconverted losslessly. void* and fn() could still have different ABIs, both for optimization (for example: If you had a register dedicated for fast subroutine calls and the abi put the first fn-ptr parameter/return value in that register) and for runtime signature verification purposes.
My point is that as far as program behavior goes, the information lost by this cast is invisible in most circumstances and adding these cast never breaks code, so describing it as undesirable because of being lossy is misleading.
This is not reasonable; one or more of these optimizations is just invalid (probably only because rust has no TBAA). One possible solution is that "we replace the *storage_ptr by the value that has just been written there" is replacing exposed-provenance (p+q) with p-provenance (and possibly the first optimization needs to maintain the escape of q, much like other optimizations need to maintain inttoptr(ptrtoint(x))).
People mentioned this in that bug, and while you said PVNI does not cover it (because after all spec C as TBAA), it does have transmute-via-union, and explicitly calls this out said type pun as an exposure.
Now rust can make its own model, but C has a solution, and LLVM will therefore support said solution (and will likely support exactly this case since clang -fno-strict-aliasing exists). On top of that rust has loudly promised no TBAA; having TBAA in this special case and then being even more restrictive than C (since you're extending this to all transmutes, not just pointer aliasing) is extremely questionable. And claiming it is necessary is just untrue.
The claim that a (heavily optimized) language is a lot simpler to specify precisely if you replace int-to-ptr casts by the operation I described above is not misleading, it is just a fact.
That int-to-ptr casts are undesirable is not misleading, it is my opinion, mostly based on the observation I just stated. You picked "lossiness" as a strawman, but that was just one point in my list of arguments -- it is a good starting point because most people intuitively think it is not lossy, so explaining this is the first step to pointing out the subtleties of provenance. You seem to be fine with it being lossy, and still consider int-to-ptr casts a reasonable operation -- that is fine, I disagree but respect that position. But don't be surprised if the precise spec then looks more like Cthulu than anything civilized.
This is going waay off-topic, so let me just say that no, C does not have a solution. They have some patches for some of the easier cases, but no general solution has been proposed yet. TBAA helps paper over parts of the problem, but LLVM does not have TBAA (or rather, it is opt-in), so this will not help LLVM either. Also, no idea why you claim Rust would be "more restrictive" than C, that is not the case.
Sure, you could fix this by optimizing less, but I know many people that would call this "not reasonable". If you combine my two examples, you will see that removing a load whose result is unused is an incorrect optimization. I very much doubt any compiler author will be convinced to give up that optimization. (The only alternative is to say that integers have provenance, which then has a whole other set of "unreasonable" consequences.)
But this thread is about CHERI, w65, and usize vs ptr_t, so I suggest that we keep the discussion related to that. To recall, the reason I brought up these provenance issues in the first place is that I think that replacing int-to-ptr casts with an operation that explicitly accounts for provenance would solve most of the issues around CHERI. I personally happen to think this is a good idea independent of CHERI, but I don't expect everyone to agree and that should have little bearing on this thread -- it remains a useful option for CHERI support and might be less invasive than some of the other proposals.
The part that is relevant to the discussion of CHERI and usize/size_t is that loss of CHERI's extra pointer bits is a very different sort of loss than losing provenance information. Without CHERI's extra bits you can't use the "pointer"; it's not really a pointer anymore. Losing provenance information via a method the compiler understands still leaves you with a completely valid pointer. (And if you're proposing changing /that/, tbqh it's way too late)
Whilst true, it's worth noting that CHERI's capability metadata does represent provenance, but at run time. It's not exactly the same as the compiler's provenance model, but in Rust it could actually be close enough to be useful. Any experimentation in that area requires the usize question to be resolved, of course.
If your goal is solely "manipulate low bits in pointers" then sure, this is a kludgy way to do it. But that's not the only way people use uintptr_t. People love to throw them around as union { size_t; void *; } equivalents and pass them all over the place. If you suddenly need to pass around a pair instead of uptr then you will take a bunch of overhead for doing so on all targets.
So, I see this is a workaround for specific use cases that don't solve the underlying problem.
The API is also slightly awkward for CHERI. If you give it the original pointer you cast to usize then it can work fine. But in our C/C++ work we have opt-in subobject bounds support (and the aggressiveness is configurable; as you turn it up it does break some parts of the C language, but that is a necessary evil for mitigating intra-object overflows), and if that is mirrored in Rust (which is presumably much easier to do than C due to the borrow checker relying on things like that) then there is the potential to get confused about which pointer should be used for provenance and accidentally amplify the bounds (i.e. you pass a pointer to the struct rather than a pointer to the field in the struct).
Thus, given that your proposal is not general enough and provides sharp edges for CHERI, I don't think it's a good idea.
Indeed, CHERI lacks the arcane trickery that is required in a proper spec of Rust to magically construct usable pointers from a usize. So one way of dealing with the issue is to move people towards programming patterns that do not require this trickery in the first place. I think this is perfectly possible, and Rust will be better for it even outside of CHERI.
Yes, in C it is very common to use integers as type for "int or ptr". I am not sure how common this is in Rust. IMO it is a bad call, and it would be much better to use pointer types for this purpose. If you do that, then the proposed API, together with a "null provenance" that construct non-dereferencable pointers just to store a usize in a pointer, is all you need.
This is an argument for my API, not against it, since that API avoids any guesswork -- the program is always very explicit about which subobject the pointer may access.
In fact, Stacked Borrows already enforces tight subobject bounds on references. And if Rust had ptr_from_int instead of int-to-ptr casts, Stacked Borrows would be a hell of a lot easier to define. All that business around "untagged" pointers could be removed. "Untagged" entirely breaks some otherwise nice properties Stacked Borrows has, and I am not sure if there is a good fix here.
But the upshot is, subobjects are basically trivial with my proposed API: the resulting pointer can access the same subobjects as the supplied provenance.
(Moreover, the strict subobject enforcement of Stacked Borrows is one of its most-criticized aspects, so chances are Rust does not want those strict subobject rules to begin with. I will try to do away with them in a future version of Stacked Borrows. Then this point would be moot anyway.)
Sure, it's annoying practice, but people do such things. You can make it work, but you cannot eliminate the additional overhead from carrying two values around rather than just one.
It "works". But that is not my point. Clearly if you provide the original pointer you cast to usize then everything works just fine. But that requires you to do that correctly and understand the subtleties of subobject bounds. It provides a means for you to accidentally amplify the bounds of your pointer compared with the original one you cast to usize if your programming style is sloppy. Contrast that with just defining a uptr that is a real pointer, where the bounds are carried all the way through and doing so is impossible unless you deliberately do something to amplify bounds (i.e. you use something like your proposed little function and bypass uptr), which would stand out like a sore thumb as being dubious code, compared with your proposal where such patterns would be forced. For example, you could accidentally do the following (in C, because I would surely butcher Rust syntax with my lack of experience):
struct S {
int x;
char buf[64];
};
char *
aligned_buf(struct S *s)
{
size_t buf_addr = (size_t)s->buf;
size_t aligned_addr = (buf_addr + 15) & ~15;
return (ptr_from_int(s, aligned_addr));
}
That would give you a pointer whose bounds cover the whole struct, as ptr_from_int should have used s->buf not s. With uintptr_t you can instead write:
This is shorter, simpler and will carry the subobject bounds all the way through, so the returned pointer's bounds will only cover s->buf. If you make people write the former, people will get it wrong, especially when the usize cast and ptr_from_int call are not next to each other, as it's all too easy to screw up. Especially since the failure mode for getting it wrong is not an error/crash, even on CHERI (unless the pointer you use doesn't encompass that address, has too small bounds, etc, only talking about when you accidentally use an overly-permissive pointer), instead you just silently continue with a too-permissive pointer floating around that could have security implications, depending on what you use it.
I am not sure what happens in CHERI, but with regular C/Rust provenance, when you cast a pointer to an integer and back, your provenance does get amplified. That is the only way to make all this existing code working, given the fact that the conversion loses the exact provenance: when doing an int-to-ptr cast, we have to construct a conservative overapproximation of the provenance that the pointer might have had.
If CHERI maintains the exact provenance here, then it is actually incorrect in the sense of ruling out valid Rust (or C) programs that rely on the fact that doing a ptr-int-ptr roundtrip loses some precision in the provenance.
This (using the restricted bounds) is, in general, actually wrong, though -- the provenance of the underlying pointer should get lost, because the pointer was round-tripped through an integer, which necessarily loses provenance precision. Programmers should explicitly maintain provenance through such a roundtrip, instead of relying on the conservative overapproximation of provenance that will happen otherwise.
Put differently: if this roundtrip maintains provenance, then many optimizations that compilers regularly perform are just incorrect (as my previously mentioned blog post explains). Now, you could try to salvage this by saying that it maintains provenance in C but not in LLVM IR, since the optimizations are performed on the IR and hence they do not have to be valid transformations in the surface language -- but I am not convinced that this is a good idea, at least for Rust. It means we have to come up with some ridiculously complicated semantic model explaining how pointer provenance is preserved here, just so that all this provenance can be thrown away when Rust is lowered to LLVM IR (or probably even already when it is lowered to MIR, because we want to do more and more optimizations on MIR eventually).
(And note that even for C, the PNVI proposal does not exactly preserve provenance in all cases. When you cast a one-past-the-end ptr to an integer and back to a pointer, you may in some circumstances use it to access the first byte of the following object. It sounds like CHERI would already reject some well-formed programs here. Arguably nobody cares about those programs to work, but things get a lot more tricky when you consider restrict, which the current provenance proposals AFAIK do not do.)
