On Casts and Checked-Overflow

internals bikeshed (deprecated)
1

On Casts and Checked-Overflow

The RFC 560 text includes the following new error condition (with associated "defined result"):

  • When truncating, the casting operation as can overflow if the truncated bits contain non-zero values. If no panic occurs, the result of such an operation is defined to be the same as wrapping.

This raises a few questions:

  • How is the sign-bit treated? E.g. Is the sign-bit of a negative input value considered a truncated bit when e.g. casting an i8 to u8? (Note that this implies that -1_i8 as u8 can panic.)

  • Are the truncated bits interpreted directly, or are they logically-inverted for a negative input value? For example, one might interpret the above text as saying that -1_i16 as i8 can panic, since -1_i16 == 0xffff_i16 which is non-zero in its upper-eight bits.

As part of effort towards finishing off the Arithmetic Overflow Tracking Issue, I have spent some time working through the space of possible interpretations of this text, and have identified three potential interpretations that are each independently useful.

The goal of this post is to describe those three interpretations, and to provoke a dialogue about what interpretation (either of the three, or perhaps another I had not considered) Rust should use for overflow-checking of cast operations.

NOTE: Throughout this text I use literals, written in hexadecimal, in my examples; but the point is that you should think of them as representative for some code that just says <identifer> as <type>, and I happen to be telling you what the value and type are via the numeric literal syntax, using hexademical to make it clear which bits are set to nonzero values.

Goals and Examples

The goal of the overflow-checking is to catch bugs: Cases where the programmer has not clearly expressed their intention, and may be inadvertantly throwing away important state in the cast operation. At the same time, we do not want to introduce an undue burden when writing casts that are "intuitively valid."

So for example, here are some simple cases where it seems useful to trigger a panic :

  • 0x102_i16 as u8: an unchecked-cast would yield 2, throwing away the high order bit. (This seems like the very definition of a dangerous truncation; consider e.g. casts from u64 to usize on 32-bit targets.)
  • 0x8003_i16 as i8: again, an unchecked-cast would yield 3, throwing away the sign-bit (and the fact that this is a negative value of large magnitude).
  • 0xFFFE_u16 as i8: this case is analogous to the previous one; here an unchecked cast would yield -2, when the original input value was 65534.

But here are some examples where at least it is not so clear cut whether a panic is warranted (if not outright obvious that we should not panic):

  • 0xFFFE_i16 as i8: this is a cast of -2.
    • It seems like this should be entirely safe to cast to i8; but as noted in the questions above, ensuring we do not panic means we need to not treat the higher 8 bits as truncated here.
    • Note also that the input bit pattern here is the same as the one we gave in the third example above of where panic seemed warranted -- so only the type of the left-hand side is what makes the difference here.
  • 0xFF_i8 as u8: this is a cast of -1 to a range that cannot represent -1. However, I think one quite frequently encounters cases where one casts directly to the unsigned counterpart of a signed type in order to e.g. be able to do logical right-shift on the bits (i.e. shifting-in zeroes rather than the sign-bit).
  • 0x81_i16 as i8 : this is a cast of 129 to a range that cannot represent the value 129; but one can interpret the highest order bit as a sign-bit, yielding the signed value -127.
    • So, in one sense, no bits of information have been lost (and thus there has been no truncation).
    • But in another sense, the denotation of the value has been completely changed, and thus perhaps a panic is warranted.

Terminology

In the text below I use some technical phrases, which I will define here:

  • An integral type is one of the types iN or uN for some N in {8, 16, 32, 64, size}
  • The bitwidth of an integral type iN or uN is N. (Note that the bitwidth size is considered distinct from both 32 and 64, regardless of the target architecture's address size.)
  • The phrase "the signed version of I" (for some an integral type I = iN or uN) denotes iN
  • The phrase "the unsigned version of I" (for some an integral type I = iN or uN) denotes uN
  • Unless specified otherwise, t is some integral type.
  • Unless specified otherwise, x is an identifier that has some integral type (which may or may not be equal to t).
  • The phrase "The mathematical value of x" means the value of x when interpreted (according to its type) as an signed integer of arbitrary precision. Thus:
  • the mathematical value of 0xFF_i8 is -1
  • the mathematical value of 0xFF_u8 is 255
  • the mathematical value of 0x8000_i16 is -32,768
  • the mathematical value of 0x8000_u16 is 32,768
  • The phrase "x falls in the range of t" means the mathematical value of x falls in the closed interval [min,max], where min and max are the mathematical values of t::MIN and t::MAX

The Three Interpretations

So, with that in mind, here are the three interpretations I have identified:

  • "Strict Range" - x as t may panic unless x falls in the range of t.
  • "Width Oriented" - x as t may panic unless either
    • the bitwidths of the type of x and t are equal, or
    • x falls in the range of t
  • "Loose Range" - x as t may panic unless either
    • x falls in the range of the signed version of t, or
  • x falls in the range of the unsigned version of t

(There may exist other interpretations of the text beyond these three, but these were the ones that I identified that seemed potentially useful.)

Some examples:

  • All three interpretations say that -1_iN as i8 can never panic, for any N, because -1 falls within the range [-128,127].
  • "Strict Range" and "Width Oriented" both say that -1_i16 as u8 can panic, since -1 falls outside the range [0,255]
  • "Strict Range" says that -1_i8 as u8 can panic, for the same reasoning as above.
  • However, "Width Oriented" says that -1_i8 as u8 can never panic, because the bitwidths of the input and output types are equal. (Note that this implies that -1_i16 as i8 as u8 can never panic, even though -1_i16 as u8 can.)
  • "Loose Range" says -1_iN as u8 can never panic (for any N), since -1 falls in the range [-128,255].

No two of the three interpretations are semantically-equivalent; for any two interpretations, there exist inputs where the panic behavior may differ (as illustrated in the examples above).

Comparison

Let's assume that one of the above three interpretations is the one we desire. The question is: Which one?

My current preferential ordering (most preferred first) is:

  1. "Width Oriented"
  2. "Loose Range"
  3. "Strict Range"

I believe a good solution must support -1_i8 as u8; at the same time, I think Rust should be allowed to panic on 129_i16 as i8 (more on this latter point in a few paragraphs).

"Width Oriented" is at the top because it satisfies both of the previously listed conditions, and general, appears to have desirable behavior (see the illustrative implementation linked below).

"Strict Range" is at the bottom: I think 0xFF_i8 as u8 should not be allowed to panic; we need to make it easy to do bit-oriented computations between types of equivalent bitwidth especially when its not losing any actual bits of information.

"Loose Range" is in the middle because it beats "Strict Range" (by supporting -1_i8 as u8), but it is not at the top because I think it is strange.

  • At first I thought "Loose Range" was a strong contender because it seems very uniform. But consider the cast 0x81_i16 as i8 (aka 129_i16 as i8): the "Loose Range" interpretation allows this (i.e. will never panic), and converts the value 129 to -127. The "Width Oriented" interpretation, on the other hand, allows a panic to occur here, since 129 falls outside the range [-128,127].
  • Note that "Width Oriented" allows a panic on 0x81_i16 as i8 and 0x81_u16 as i8 but forbids panic on 0x81_u8 as i8; in all cases the input value is 129, but the relevant difference is the type. We consider it safe to do the u8 to i8 cast, because we assume that the matching bitwidths indicates that the reinterpretation of the sign bit-is intentional.
  • Another way to look at this whole situation is that the "Width Oriented" avoids a truncation of the sign-bit in such a case.

Illustrative Implementation

The following linked gist has some code illustrating the three strategies and their behavior on various boundaries cases when casting to i8 or u8, as well as a transcript of the code running.

2

I think when not otherwise specified, it should always be the logical content that is considered, and not the representation. So a cast should be valid if and only if the casted value is within the range of the casted-to type. (This is your “Strict Range”, I think.) I think this is the spirit of our overflow-handling design (why should -1i8 as u8 be valid and not 0u8 - 1?), as well as of some other ideas for improved ergonomics.

Last I saw, the WrappingOps trait had grown, or was intended to grow, methods such as wrapping_as_u8(), wrapping_as_i8(), wrapping_as_u16(), and so on for each primitive type. I think these are what should be used when anything other than “Strict Range” semantics is desired. (Whether there’s a way to make these less monomorphic and more ergonomic, e.g. some kind of wrapping_as() that’s generic over the target type, is an independent question, I think. Also perhaps we could make foo as Wrapping<u8> where foo: Wrapping<i8> work in the future, having the old semantics of as on the primitive types.)

1 Like
3

Hm, I always assumed "Strict Range" to be the intended semantics. -1i8 as u8 -> Can u8 represent -1? No. -> Panic! Just like with all the other operations. I.e. a clear separation exists - all the operators work on integral numbers in mathematical sense, and all the bit twiddling with representations is done with non-operator methods (at least for now). Regarding ergonomics, the whole RFC is a big hit to the ergonomics with as being just a particular case.

4

Of these options, I very strongly prefer “String Range” semantics. My understanding of the point of the RFC was to treat numbers mathematically unless the user opted into bitwise semantics (wrapping, etc.). Making as an exception to this would be very inconsistent. Thus, if the logical value of x cannot be represented by the target type, it should be a panic. This should definitely include -1_i8 as u8. Similarly to wrapping, if the user wants to explicitly cast the bit representation and not the logical value, there should be a different way to do that.

5

Agree with strict, with the caveat that not having a builtin way to truncate, or to bitcast between signed and unsigned, for 1.0 would be really silly. (Of course you could do both manually, but still-)

6

As others have already said, I expected the behavior to be like this:

Let x contain an integer v (whether it x is a variable, typed, or untyped literal does not matter) and T an integer type. The behavior of x as T is as follows:

If v can be represented in T, the integer contained in x as T is v. Otherwise a panic occurs or the integer contained in x as T is computed by the following rules:

  1. If T is an unsigned type, the maximum integer that can be represented in T is repeatedly added to or subtracted from v until the resulting integer can be represented in T.

  2. If T is a signed type, twice the minimum integer that can be represented in T is repeatedly added to or subtracted from v until the resulting integer can be represented in T.

This avoids any platform dependent formulation, only deals with abstract values, and treats integer types as containers that can contain certain integer ranges without specifying the way the integers are represented.

I believe a good solution must support -1_i8 as u8;

I do not think so. If you want the maximum value that can be represented in u8 you should use !0. This is not an arithmetic expression and thus not affected by integer overflow semantics.

I think 0xFF_i8 as u8 should not be allowed to panic; we need to make it easy to do bit-oriented computations

Signed integers should not be used as bit containers. The semantics are unnecessarily complicated. isize as usize and usize as isize should definitely be allowed to panic because they are used often with pointer arithmetic and using the wrong value there can lead to UB.

1 Like
7

I think that if narrowing-conversions are going to panic they should panic when they can’t represent the target correctly (strict semantics, i.e., that ((x: A) as B as isupermax == x as isupermax never returns false), but that also there should be easy access to the epimorphisms.

This makes 0x00FEu16.project::<i16>() as i8 == -0x02i8 problematic, because 0xFFFEu16.project::<i16>() as i8 == -0x02i8 is also true - we do lose the “sign-extended” most significant bit.

8

What is this project method of which you speak?

9

Sorry. .project is The epimorphism/projection/wrapping cast.

10

Okay, so ... I guess I'm confused.

When you said

This makes 0x00FEu16.project::<i16>() as i8 == -0x02i8 problematic ...

what was the "this" that you were referring to? Was "this" the "Strict Range" interpretation, as I had thought based on the context of the above quoted statement?

Because I do not see the problem you are referring to ... isn't 0x00FEu16.project::<i16>() == 254, and therefore, since 254 is not representable in the range [-128, 127], a panic would be allowed during the evaluation of the as i8 there?

11

A thought is that we use as specifically to indicate a type conversion which is dangerous, and yet the strict range (which I agree with the comments is the ‘obvious’ choice) is making it safe. So perhaps we should be using coercion for safe conversions using strict range and as should keep its current semantics of being a memory cast? The obvious problem is that when we build without overflow checks we have dangerous, implicit coercion which is very bad ™.

However, it does seem to me that a systems language restricting ‘dangerous’ casting to ‘numerically’ safe operations is also very bad. I feel like integers are almost as likely to be used to represent bitsets or other non-numeric data as they are to represent numbers and so not allowing ‘unsafe’ casting is too restrictive.

12

I don't think "dangerous" is the right word for casts by as. Maybe I would use a word more like "nontrivial" or "nonobvious". (If it were actually dangerous, then it should have a much less innocuous name.)

yet the strict range (which I agree with the comments is the 'obvious' choice) is making it safe

Not really - I wouldn't say that panicking is all sunshine and kittens.

13

That would be still bad - even the strict-range narrowings are not particularly safe, as they can panic.

14

I’d suggest to only allow the casts falling into the strict range of the target type (as integers, not as bits or anything).

If the code author’s intent is to have truncating conversions, then being explicit about it would be very nice. In today’s Rust, as u32 or similar are leniently used where it really doesn’t mean bit truncating, because it’s such a short way to write it (shorter than the more correct .to_u32().unwrap()).

I think we’re facing something similar to the uint -> usize rename here, which also resulted in the “non-obvious” thing (uint not being the default, as not being a normal cast, but rather a bitcast) being renamed, so people don’t accidently assume it’s the right thing to do (renaming to usize, making an explicit method for bitcasting).

15

I initially leaned towards the strict interpretation as well, but after the discussion at the meeting, I am starting to feel conflicted. The major points made there were:

  1. Bitcasting is far more common in practice (unsubstantiated, but see below), so this could be annoying.
  2. Subtle rules may be more confusing than helpful. Probably as should either be infallible (as today) or strict.

I tried to do a brief survey of rustc to uncover the kinds of places we use as. This is not a super thorough survey and in particular I didn’t count instances (and we’d want to look at more than rustc, ideally). But in any case here is a listing of what I saw. These are reasons to use as, somewhat sorted by frequency of occurrence (but like I said, I didn’t count).

  1. To convert between usize and u32, usually for indices whose length is believed to be bounded to u32. In this case, I think that the “width oriented” approach is probably best. That is, you’d prefer a panic if converting from usize to u32 throws away some high-order bits when debugging.
  2. To convert from u32 to i32, for example in trans when converting offsets for use with a GEP statement. Here, strict is preferred.
  3. To upcast, in which case any interpretation is fine.
  4. Constant evaluation sometimes goes back and forth between i64/u64. I think in this case strict is preferred.
  5. To convert from i32 to u32 in binary encoding. Here, width, loose, or truncating is preferred.
  6. To select individual bytes ((x >> 16) as u8). Truncating is preferred.

One thing I did not see was a converting from i32 to u32 where we wanted to ensure that the value was not negative at that time. This probably does come up, but I don’t think I saw a case of that.

Anyway, here are some further concerns:

  1. If we keep as as it is, which I think is reasonable, then it does raise the question of whether we will offer stricter versions. If so, will they be tied to methods? It is perhaps inconsistent to say that + is strict by default but as uses a method. But there is clearly utility for at least some kind of checking to ensure we don’t throw away bits.
  2. If we convert as, it may be annoying.

This is connected to an existing mildly open question on how to make it more ergonomic to use wrapping operations. I am personally leaning towards something like C#'s checked blocks. Basically allowing you to say that you want wrapping semantics within a particular fn or block (but not module, I’d prefer to keep the annotation local). This would cause all lexically enclosed + operations to map to a WrappingAdd trait – and would also affect primitive operations. I’ve been meaning to write a separate post sketching this out. Anyway, having a convenient way to switch the interpretation of as may make the default stricter interpretation more appealing, though I feel like as is a bit different than + in that a series of wrapping operations often come together, but as conversions are frequently isolated (citation needed).

Overall, I feel conflicted! Usually when I feel conflicted, I lean towards the more conservative option, which suggests keeping as as infallible. Perhaps data is the key to making this decision.

16

As with overflows and wrapping, I think the most important thing is that the intended semantics - numeric cast vs. bit-cast - should be clearly differentiated, whichever one winds up with the nice as syntax.

(My personal preference is that as should be for numeric casts and we should have a trait for safe transmutes (the old Coercible/Transmute/ReinterpretCast idea) which would be used for bit-casts. Of course, the latter can also be solved in a narrower way.)

17

I feel like the current situation isn't as good as it could be: If you want bitcasts, you use as. If you want numeric conversions and you use as you introduce a subtle bug.

If you consider the opposite situation, I think it looks better: If you want numeric conversions, you're fine with as. If you want bitcasts and use as, you're going to see your code panic and realize you made a mistake.

I think the more conservative (less bug-inducing, "safer") option is the latter one, unless you're talking about conservative as how things were in the past.

1 Like
18

I meant conservative as in less change, closer to the current known state.

19

I collected some statistics not so long ago: http://internals.rust-lang.org/t/the-operator-as-statistics-of-usage/1353 It looks like most of numeric conversions happen in “mathematical” and not “bitcast” context and should be checked.

20

Great, these sorts of statistics are good, but I'm not sure I draw the same conclusion. It's a bit hard to say though. I'll have to go through and categorize your examples -- but some of them, like x as u32, it's hard to know how to categorize without more context. Basically I think I want to draw up a table to know what percentage of uses would be permitted by the various possible interpretations of as.

I do suspect that if we leave as the way it is, then it basically serves well as a low-level, type conversion operator, but we'll find that we recommend using other methods of casting (e.g., methods) for converting between integers in safe code in particular. It seems to be unusual that one really wants to drop bits on the floor; but not necessarily unusual that one wants to reinterpret bits as a new number (which seems to be the dividing line between the strict interpretation vs the others). The appeal of one of pnkfelix's hybrids seems to be that you give up very little -- you can get today's semantics by adding a mask, and you get safety checking in a number of common cases. But they are more complicated, and they definitely remove as from the realm of "numerics" and into the realm of "bits".