[Pre-RFC] Another take at clarifying `unsafe` semantics

[H2CO3]

Alternatives

1. Do nothing. Unsafe abstractions will remain confusing to implement and review, making unsafe Rust hard to learn and causing memory, type and thread safety issues that could have been avoided by improving unsafe code ergonomics.

Well, that’s quite an exaggeration. What’s hard for one person might be trivial for another. The unsafe keyword is, in my opinion, the smallest ergonomic problem in today’s Rust. Adding pre and post won’t suddenly enlighten all the programmers coming from high-level languages. It seems to me that you are significantly overestimating the problem.

If there is a problem at all — I don’t feel at all that the current unsafe design has any clarity issues. Again, inevitably, there will always be newcomers who find certain concepts hard. This cannot possibly be fixed by just piling more syntax on top of what we already have — they will have to invest significant effort into learning new concepts such as unsafe code either way, and I would argue that adding more complexity to the language wouldn’t help, conversely, it would impede their learning experience.

[HadrienG]

I think part of our opinion divergence here lies in the fact that we are not looking at the same side of the issue.

From my understanding, your comment asks for a guide to all the existing uses of unsafe constructs in the Rust language and standard library. What can go wrong, how to avoid it, together with some attempts at organizing these problem/solution pairs.

In constrast, this pre-RFC is about the tools used by Rust developers to create new unsafe abstractions. Basically, any time the unsafe keyword is used in someone’s code for any other purpose than introducing an unsafe block.

The former issue is, as you say, largely a documentation issue, which I believe is targeted by many existing projects. The latter reaches beyond that, in the sense that Rust’s current vocabulary for building unsafe abstractions is unnecessarily ambiguous and confusing even when you are familiar with it, due to the way it mixes together unrelated notions such as…

• Unsafe blocks (“where bad things can happen”)
• Unsafe preconditions (“what you need to do to prevent it”)
• Unsafe postconditions (“how we help you at the task”)

Addressing this language-side issue is what this pre-RFC is about. From this perspective, I believe that what you are discussing is not in conflict with what I am proposing, but rather an important orthogonal issue.

[Centril]

Conversely, I understand that dependent types would be a much larger addition to the language than what this proposal suggests

A limited form of dependent types are already being added to the language, see const generics.

Dependent types have great power / complexity ratio.

[HadrienG]

Well, that’s quite an exaggeration. What’s hard for one person might be trivial for another. The unsafe keyword is, in my opinion, the smallest ergonomic problem in today’s Rust.

I would not want to engage in a detailed comparison of Rust’s ergonomic issues, but considering the amount of time and discussions it takes even highly experienced Rust developers to figure out the basics of what can and cannot be done with pointers and references in unsafe code, I think we can safely (no pun intended) say that unsafe semantics are not trivial for anybody

This is why we have many projects which aim at clarifying these semantics, to which this pre-RFC is only one contribution in an area that was IMO insufficiently explored so far: the role of language semantics and ergonomics in unsafe abstraction design.

Adding pre and post won’t suddenly enlighten all the programmers coming from high-level languages.

…and thankfully, this is not the goal of this pre-RFC. Which is an attempt at clarifying the semantics of the tools that are used to build unsafe abstractions, a rather advanced Rust task which I do not expect complete beginners to engage into.

It seems to me that you are significantly overestimating the problem.

That might be the case. However, the amount of people who are constantly confused about what is perhaps the most dangerous feature of the Rust language worries me.

In the world of life-threatening systems, the more dangerous a tool is, the harder you make it to use it incorrectly. Knives have a handle shaped in such a way that it is pretty clear how you are supposed to pick them up. And power tools are similarly designed in such a way that you are unlikely to have your hand on the cutting bit when you turn them on.

Now, contrast this with Rust’s unsafe abstraction vocabulary:

• Unsafe Rust should never trust Safe Rust’s abstractions, yet there is no safeguard in place to lint against reaching for these, in spite of this being a likely habit of the typical Rust programmer (who writes a lot more safe code than unsafe code).
• Trait’s unsafe associated methods are, in current usage, solely about preconditions. All this usage of “unsafe” states is that the method’s implementation can rely on a certain property to be true for memory safety, but are not obliged to. Yet the method’s entire implementation is automatically turned into an unsafe block, silently enabling all kind of unsafe operations without any compiler warning/lint.
• TrustedLen is another example of unsafe abstraction design gone wrong. This trait silently changes the semantics of another trait’s methods in a way that affects memory safety. Concretely, what this means is that the only thing keeping your unsafe code from doing something dangerous is a single trait bound. If you forget that “TrustedLen” bound anywhere, then you will open a memory safety hole without any warning. This would not have happened if, in an example of more principled abstraction design, the trusted length was provided as a method that is specific to the TrustedLen trait, which in turn would have required a notion akin to the proposed unsafe(post) fn.

It seems to me that there is definitely quite a lot to be done to make unsafe Rust abstractions as easy to use correctly as real-world sharp tools are And this pre-RFC proposes to start by clarifying what an unsafe abstraction is, and separating its inner components cleanly instead of the ambiguous soup that we currently have where the “unsafe” word is used to mean so many different things.

If there is a problem at all — I don’t feel at all that the current unsafe design has any clarity issues. Again, inevitably, there will always be newcomers who find certain concepts hard. This cannot possibly be fixed by just piling more syntax on top of what we already have — they will have to invest significant effort into learning new concepts such as unsafe code either way, and I would argue that adding more complexity to the language wouldn’t help, conversely, impede their learning experience.

See above for some examples of highly perfectible unsafe design that you can play with today As with @gbutler, I think I should again clarify that I personally see two “layers” of unsafe users:

• People who use the unsafe constructs provided by the Rust language and its standard library. This happens relatively early in the process of learning Rust, for example the first time one tries to optimize array manipulations or do FFI. Doing this only requires unsafe block. I think this is the most mature part of the unsafe ecosystem today, though as @gbutler points out it could certainly use better documentation.
• People who, in addition, try to build unsafe abstractions which other developers (including future maintainers of the same crate) can rely upon. This is a more advanced topic, where one gets to invoke the other meanings of the unsafe keyword (unsafe fns, unsafe traits, unsafe impls…). These are the uses of the unsafe keyword which I think have unnecessarily unclear semantics, and could benefit from some language-level clarification.

I probably need to reword the pre-RFC in a way which makes it much clearer that I am catering to the latter population here. My wording is obviously not yet clear enough on this front.

[gbutler]

I think part of our opinion divergence here lies in the fact that we are not looking at the same side of the issue

Yes, I agree. I tend to think that the thing that is missing is guidance and education rather than syntax. That being said, I also agree that differentiating between “unsafe blocks”, “unsafe trait impls”, and “unsafe functions/contracts” more clearly is worthwhile, but, I just don’t see that “unsafe(pre)” and “unsafe(post)” (or anything like that that I can really think of even though I initially approached this topic thinking that was the way forward) will usefully accomplish that goal.

Again, these are just my opinions and I’m probably wrong, but, I’m really starting to believe that there is no special keywords that could be used instead of just “unsafe” that would make the situation any better from an understanding perspective.

That being said, this is definitely a worthwhile discussion to have and I will keep an open mind about it for sure.

[gbutler]

People who, in addition, try to build unsafe abstractions which other developers (including future maintainers of the same crate) can rely upon. This is a more advanced topic, where one gets to invoke the other meanings of the unsafe keyword (unsafe fns, unsafe traits, unsafe impls…). These are the uses of the unsafe keyword which I think have unnecessarily unclear semantics, and could benefit from some language-level clarification.

Ah…I think this comment more clearly expresses what you are going for. Definitely something to chew on.

[gbutler]

Doesn’t feel ready for an RFC. Still a little unclear on what will be accomplished for the amount of churn.

[gbutler]

unsafe(post) functions provide guarantees which can be relied upon by unsafe code. Therefore, they can only return to their caller inside of an unsafe code block. These functions can safely be used when collecting information that is to be passed to an unsafe(pre) function, such as

Unfortunately, there is no difference from being called from inside and unsafe block and returning inside an unsafe block. In both cases, the call to the function must be wrapped in and unsafe block (unless you are proposing something like???):

unsafe { let x } = fn( zzzz );

Where ‘fn’ is an unsafe(post) function??? This would seem radically unuseful and confusing (to me) though.

[gbutler]

Contrary to current unsafe fns, these modified forms of unsafe do not implicitly make the body of the function an unsafe block. The rationale for this is that only the author of the function knows which parts of the function, if any, are unsafe.

This I like. I agree that mixing up declaring an unsafe contract with and unsafe block of code is a mistake in the current implementation.

[HadrienG]

It is not unsafe to call an unsafe(post) function. All this annotation tells you is that you can rely about some properties of the result (described in the function’s documentation, like all unsafe contracts) for memory safety. If you are not doing unsafe things, then you can safely ignore the annotation.

In this sense, unsafe(post) is like const fn: a contract which sets requirements on the implementation, which the user may or may not rely on.

This is very different from unsafe(pre), which sets requirements on the user, which the implementation may or may not rely on.

[gbutler]

They all guarantee properties which unsafe code is allowed to rely upon, and which could be violated by a careless implementor, resulting in unsafety.

Is this true? Unsafe code is specifically not permitted to rely upon safe code being implemented correctly for correct behavior (lack of UB). Should unsafe code be allowed to rely upon other (far distantly implemented) unsafe code to have been implemented correctly in order for it to not have UB? This doesn’t “FEEL” right. I think this needs some exploration.

[gbutler]

It is not unsafe to call an unsafe(post) function

Then, why:

Therefore, they can only return to their caller inside of an unsafe code block

[HadrienG]

Is this true? Unsafe code is specifically not permitted to rely upon safe code being implemented correctly for correct behavior (lack of UB). Should unsafe code be allowed to rely upon other (far distantly implemented) unsafe code to have been implemented correctly in order for it to not have UB? This doesn’t “FEEL” right. I think this needs some exploration.

Send and Sync are two examples of prior art. Unsafe code is allowed to rely on Send types being safely sendable across threads, and on Sync types being safely shareable between threads, in spite of the Send and Sync implementations for the type being defined kilometers of code away.

[HadrienG]

It is not unsafe to call an unsafe(post) function, but it is unsafe to implement one, because a bad implementation of an unsafe(post) function will cause memory/type/thread unsafety. This is the same reason why implementing Send and Sync is already unsafe today.

The rule which I found to formalize this requirement in a way that is symmetrical with unsafe(pre) is to mandate that these function’s return statements be within an unsafe block, and thus characterized as potentially unsafe operations, like this.

unsafe(post) fn trust_me() -> usize {
    /* Do something inconsequential */

    // This is the part where it gets dangerous
    unsafe { 42 }
}

The rationale is that return statements in unsafe(post) functions propagate safety-critical information to the outside, and are thus unsafe operations. But I’m not 100% satisfied with this rule, and am open to bikeshedding it. As an even more minimal proposal, perhaps the unsafe(post) annotation on the function implementation is enough?

[gbutler]

OK, I think I’m able now to more clearly clarify my discomfort with this proposal and offer a possible enhancement that I believe could make all of this more worthy of the churn it imposes. I can best sum it up as, “Spell out the contract!”. What I mean by this would be to include some meta-language for specifying pre-conditions/post-conditions, that would server 2 purposes:

1. Clearly document the contract requirements in a way that can both be consumed by documentation generation in a standard fashion, AND
2. has the POTENTIAL to at some point be statically solvable and/or run-time optionally (debug/test only perhaps) enforced

A quick straw-man syntax/idea might be something like:

unsafe ( pre: assert ( ...some predicate DSL... )
             post: assert ( ...some predicate DSL.... ) )
fn ( .... ) -> Result

The “some predicate DSL” could be a mini, extensible language that allows expression conditions about the inputs and outputs. It could even allow an “Escape Hatch” like clause that is effectively, “We have no way of expressing this condition in the DSL, but, here is some normal human language that describes the necessary contract”.

That way, we can gradually evolve this DSL to support more and more of the kinds of necessary predicates and eventually work towards linting or otherwise statically analyzing whether or not they have been met.

Again, this is all very, very straw-man at this point and I would have to put significant effort to clarify further (I realize). I’m not an expert on this sort of thing, but, if I had my druthers, this is the direction I’d like to see pursued.

Perhaps this is an entirely orthogonal concern to what you are proposing though.

With this DSL idea, the compiler could parse the DSL predicates, determine (at some future point) if it has the potential to statically solve the predicate, if it CAN solve the predicate, and in the context of the call to the function it is able to verify that the predicate has been met, an “unsafe” block around the call isn’t required. However, if it can’t (yet) verify the predicate, it could spit out an error if no unsafe block is present and display the predicate that it can’t verify. If the predicate includes the “Escape Hatch” predicate, then the compiler obviously can’t verify and will output the the predicate for sure with an error if the unsafe block isn’t present.

If it can’t verify the predicate, but, and unsafe block is present, it would simply output a NOTE/REMINDER (like a WARNING, but, not classified as a warning) that simply reiterates the necessary predicates.

If the call is inside an unsafe block, and the predicates CAN be verified, it could warn about an unnecessary unsafe block.

Why this last? Because, then you will get an error in compiling if you break some code elsewhere that messes up the previously verifiable predicate.

To clarify further, with these predicates there are several possibilities for the compiler to find at the site of a call to an unsafe function (similar argument applies to unsafe returns):

• predicates not specified - unsafe block required/lint on function def advising to add preds
• preds specified, but, not statically solvable (aka compiler support lacking, escape hatch predicate used) - unsafe block required for call
• preds specified, statically solvable (with one of the following possibilities):
  • compiler can prove at call-site that predicate not met - COMPILER ERROR
  • compiler can prove at call-site that predicate is met - UNSAFE block not required and is NOTE/REMINDERED against (on second thought, prob should be a compiler error as well – I’ll explain below)
  • compiler cannot say for sure whether or not the predicate is met - UNSAFE block REQUIRED with output of a NOTE/REMINDER of what predicate(s) are being unsafely relied upon

I think the middle sub-bullet point above is the interesting one. Because the compiler can prove statically that you’ve met the predicate, it should be a compiler error to wrap in unsafe block. Why? Because that way, if you later change the code to break the predicate, you’ll get a new compiler error.

[gbutler]

// This is the part where it gets dangerous unsafe { 42 }

Ah…Yes. That makes more sense. I misunderstood.

EDIT: That being said, I agree with @jmst, “What’s th point?” What function would you not want to guarantee that its post-conditions are met?

[HadrienG]

I think this could be a possible backwards-compatible extension to this proposal, that could be added to the proposal’s list of future possibilities. It is not clear to me if it would be needed on day one, as long as we plan ahead for it.

[jmst]

I don’t think “unsafe(post)” on non-trait functions makes any sense.

Unsafe code could want to rely on the advertised behavior of ANY function defined in ANY crate, and there is no way to predict whether a given function would want to be used by unsafe code.

For instance, what parts of the standard library should be tagged as “unsafe(post)”? (the answer of course is all of it, which makes it pointless)

[H2CO3]

considering the amount of time and discussions it takes even highly experienced Rust developers to figure out the basics of what can and cannot be done with pointers and references in unsafe code

That seems to me like it stems from an orthogonal issue. Let me elaborate:

Danger is inherent to unsafe code. (Well, that statement is a bit redundant, isn’t it.) However you twist the syntax and/or the semantics of the language, writing unsafe code (viewed either from the “consumer of unsafe abstractions” or “author of unsafe abstractions” side) is always going to require more thinking and vigilance than usual. Yes, this can be true even for very basic concepts and operations. But I’m sure that is not a unique feature of programming in Rust. If you have studied some sort of scientific discipline at a (say) intermediate level, you probably very well know the feeling of sitting on a 1-line maths of physics problem for a couple of hours or days before being able to grasp an otherwise basic idea.

I don’t want to get too philosophical or generally off-topic here, but the problem is that our monkey brains are just not designed to do what we are doing with them. So at the end of the day, implying that intrinsically hard problems are only hard because we are not approaching them ergonomically enough and not because they are hard looks like a fallacy to me.

the amount of people who are constantly confused about what is perhaps the most dangerous feature of the Rust language worries me.

I understand and agree with that; what I am saying is that this is an inherent issue, and not something that can be mitigated completely or nearly completely by improved language design. Either interpretation of unsafe means “give up some safety guarantees”. That’s not going to be pretty either way for the reasons described above, and at the same time, the complexity introduced by the language changes laid out in this pre-RFC is, in my opinion, does not have a good enough investment-return rate. I’m not saying that ergonomic improvements to unsafe would be completely useless – I’m just trying to argue that the advantages they would bring to the table wouldn’t be worth the increased complexity.

In the world of life-threatening systems, the more dangerous a tool is, the harder you make it to use it incorrectly. Knives have a handle shaped in such a way that it is pretty clear how you are supposed to pick them up.

I absolutely agree with all of this too; in fact I’m generally the biggest fan of designing systems so that they have the least chance to be used incorrectly. However, and I don’t want to repeat myself over and over again, but the very purpose and nature of unsafe code renders attempts at making it safer very hard to achieve (I would even say contradictory).

Basically, what @gbutler wrote:

I’m really starting to believe that there is no special keywords that could be used instead of just unsafe that would make the situation any better from an understanding perspective.

You made three other very good points:

Unsafe Rust should never trust Safe Rust’s abstractions, yet there is no safeguard in place to lint against reaching for these.

I’m always in for more compiler warnings and lints! This, I think, is probably the easiest issue to address. Adding more rules/patterns to be recognized by the compiler or e.g. Clippy is an easy and in my opinion uncontroversial action, and we should go for it. If it’s likely wrong – warn about it!

As you can see, I’m in favor of these sorts of improvements as long as they don’t offset the balance towards just adding more stuff to the language in an unhealthy manner. A core language addition is a scene which always requires extremely careful consideration, and in 99.9% of the cases, the right thing to do with a new feature is to omit it. Compiler warnings and linter rules are nothing like it, they are basically free and completely safe (“safe” here meaning “does not break earlier code/concepts/assumptions or introduces bugs”).

Yet the method’s entire implementation is automatically turned into an unsafe block, silently enabling all kind of unsafe operations without any compiler warning/lint.

It’s not exactly silent, because it has a big honkin’ unsafe at the beginning. Still, I understand your concerns, I do think they are valid, and indeed, it would absolutely make sense to be able to mark a function unsafe without turning the entire body into an unsafe block. This is yet another change that likely wouldn’t need any sort of core language addition. It’s merely that the compiler should consider the body block safe rather than unsafe. If I recall correctly, that literally means flipping a one-bit flag in the compiler. (Maybe in several places, but you get the idea.)

TrustedLen is another example of unsafe abstraction design gone wrong. […]

That’s right too! I’m in favor of replacing TrustedLen with something more explicit and harder-to-forget/abuse. I still strongly doubt that this can only be done using, or that it is best achieved by, additions to the core language.

Finally:

People who, in addition, try to build unsafe abstractions which other developers (including future maintainers of the same crate) can rely upon.

Indeed, that puts the issue in a somewhat different perspective, although my comments about how more syntax wouldn’t help much still apply.

[gbutler]

comments about how more syntax wouldn’t help much still apply.

This is my feeling as well. No matter how much I WANT there to be a special syntax or keywords that makes all of this clear, nothing seems to fit the bill.