The Unsafe Rust Programming Language (Book)

[Gankro]

Hey all! As many of you know, my summer internship’s primary project is to try to better specify and document all the fiddly details you need to understand to correctly write unsafe Rust code.

My current primary effort for this is TURPL: The Unsafe Rust Programming Language.

TURPL is meant to complement TRPL as an advanced text. What exactly should or should not be included in TURPL is currently up in the air for me, and as such it isn’t an entirely cohesive document. Some sections are missing important things, and others are basically just curated copy-pastes of key RFCs (partly due to those RFCs just being absolutely excellent). Some information is duplicated or scattered because a lot of key concepts are pervasive and tangled, and I haven’t figured out a better “normalization”.

Still, I’ve progressed far enough that I’d really like to get some feedback on it! So… here it is! (rendered with my own bespoke rustdoc setup – didn’t want to setup rustbook)

• TURPL (hosted and rendered)
• TURPL (source)

Right now the current set of chapters are:

1. Intro (What is Unsafe?)
2. Data Layout
3. Ownership and Lifetimes
4. Conversions
5. Uninitialized Memory
6. Ownership-oriented resource management (RAII)
7. Concurrency
8. Example: Implementing Vec

The first 4 chapters try to address cross-cutting topics. This seems particularly necessary as these problems are vaguely pervasive throughout everything.

The next 3 look at more focused problems in detail. I try to focus on how to get the job done in Safe Rust first, and then look at how Unsafe Rust lets you go the extra mile when Safe Rust isn’t adequate.

The last chapter is a stub of a cumulative “bring it together” example.

I’ve purposefully excluded FFI and no_std because it seems like TRPL actually gives a totally sufficient coverage of the topics. There’s nothing really subtle there.

Anyway, I’m most concerned about missing important information and existing misinformation. In particular the lifetimes section is pretty technical, and my treatment of it is pretty… experimental. But really, the most important reason is that I know a bunch of you have a long plane trip with plenty of reading time ahead of you! I’ll be waiting on the other end

[DanielKeep]

Data chapter:

• “The former case quite simply wastes space.” I believe that should be “The latter case…”.

Lifetimes chapter:

• “The safest route is to just use a small function to ensure the lifetime is bound.” It doesn’t feel very clear as to what this means.

• "To understand it, consider a function that takes a function len that takes a function F". I think there is one too many “that takes a function” in that sentence.

• I like the explanation of how Iterator works for &mut [T]; my mental model on this was wrong, as it happens.

Conversions chapter:

• “casting from a smaller integer to a bigger integer (e.g. u32 -> u8)” This and the point after appear to be backwards.

Uninitialised chapter:

• “…ptr::write-style shenanigans with Plain Old Data (POD;” Unterminated left-paren.

---

Looks good so far. Nice work.

[Manishearth]

Regarding the tagged union problem, aliasing and mutability is far more nuanced than that.

The Problem With Single-threaded Shared Mutability - In Pursuit of Laziness

Tagged unions are the only language feature that rely on the rwlock pattern to be safe, but there are also many library features that need it.

[larsberg]

FFI could probably use more advanced treatment. The official docs ignore some topics like what to do about panic! and unwinding, especially at the FFI boundary. There’s also the Rust __morestack stuff around stack checking that you may want to turn off in Rust code hosted in a larger C program.

[Gankro]

[Lots of good points]

Good catches! Text updated.

“…ptr::write-style shenanigans with Plain Old Data (POD;” Unterminated left-paren.

It is closed, the aside continues after the semi-colon (I’m willing to believe it’s an awkward section of text, though).

Regarding the tagged union problem, aliasing and mutability is far more nuanced than that.

Tagged unions are necessary and sufficient to motivate lifetimes. The fact that this unlocks tons of zero-cost safe abstractions is just gravy. I suppose I could cover the gravy in more detail, though.

The official docs ignore some topics like what to do about panic! and unwinding, especially at the FFI boundary

I actually harassed steve to fix that just the other day, should be in nightly

There’s also the Rust __morestack stuff around stack checking that you may want to turn off in Rust code hosted in a larger C program.

Hmm, good point!

[Manishearth]

I also think that POD in the “uninitialized memory” bit should be expanded upon. Not having Drop is just one feature of many, and it’s not exactly a cause-effect relationship here. Copy implies !Drop, but Copy is not the same as !Drop.

(It’s also worth noting somewhere in turpl what exactly it means to implement Copy manually.)

In the last section in the intro it would be nice to suggest that whenever possible unsafe blocks should encompass the scope of their unsafety; so that all of the precondition-checking and whatnot is within the block itself.

[mdinger]

I’m interested in this but this section kinda threw me so I halted for the time being. I tried to give a little detailed feedback below.

Note: I don’t mind being perplexed by things that will be explained later if it’s noted where to look for answers or a brief explanation is given which assuages my perplexity.

---

As of Rust 1.0 there are exactly two unsafe traits:

• Send is a marker trait (it has no actual API) that promises implementors are safe to send to another thread.
• Sync is a marker trait that promises that threads can safely share implementors through a shared reference.

Okay so, there are unsafe traits so I get this.

All other traits that declare any kind of contract really can’t be trusted to adhere to their contract when memory-safety is at stake.

Immediately this is strange. All other traits must be safe if those are the only 2 unsafe traits. So, memory safety seems a very weak promise? Not sure. Maybe the next section will elaborate.

For instance Rust has PartialOrd and Ord to differentiate between types which can “just” be compared and those that implement a total ordering. However you can’t actually trust an implementor of Ord to actually provide a total ordering if failing to do so causes you to e.g. index out of bounds. But if it just makes your program do a stupid thing, then it’s “fine” to rely on Ord.

Whatever this is explaining isn’t obvious to me. Oh, sorta get it after more consideration but I don’t think what is written is very obvious. I’m not sure why index out of bounds is even a consideration. Seems sorta unsafe but it isn’t on that unsafe things list. How is a stupid thing any different from indexing out of bounds. Both seem bad…

The reason this is the case

This isn’t helpful as it relies on understanding of the previous paragraph.

is that Ord is safe to implement, and it should be impossible for bad safe code to violate memory safety.

So safe seems bad now along with unsafe…

Rust has traditionally avoided making traits unsafe because it makes unsafe pervasive in the language, which is not desirable.

The only reason Send and Sync are unsafe is because thread safety is a sort of fundamental thing that a program can’t really guard against locally (even by-value message passing still requires a notion Send).

I don’t get the explanation for why Send and Sync must be unsafe…

[DanielKeep]

All of the above only applies to unsafe code.

What he’s saying is that if you’re writing safe code and PartialOrd was a big fat liar about the whole “partial ordering” thing… well, your code might do weird things, but it won’t violate memory safety.

On the other hand, if you’re writing unsafe code that depends on partial ordering semantics… well, now it’s different. You might be doing raw pointer arithmetic based on that ordering, but PartialOrd is not actually trustworthy. To put it another way: someone going and messing up their implementation of PartialOrd could introduce memory safety problems.

So, you have two choices:

1. don’t use PartialOrd at all and just hard-code semantics that you can trust–this isn’t very flexible.

2. Define a new ReallyPartialOrdISwearOnMeMum marker trait and mark it unsafe on the basis that you (in your unsafe code) are going to trust whoever implemented that trait to do so correctly. If they got it wrong, there may be memory safety violations, but that’s only possible because they had to write unsafe to write the bad implementation in the first place.

Send and Sync must be unsafe because if you implement them when you shouldn’t, you can end up violating memory safety somewhere else in the program.

For some context, I was writing some code last night that I was pretty sure was thread-safe, but Send and Sync weren’t derived. So I went to go slap them on the type in question… then stopped and thought really hard about it for a while before writing down big comments justifying exactly why I believe those unsafe implementations are correct.

If nothing else, having to write unsafe is a great prompt to ask “are you really sure you’re right about this?”

[mdinger]

On the other hand, if you’re writing unsafe code that depends on partial ordering semantics… well, now it’s different. You might be doing raw pointer arithmetic based on that ordering, but PartialOrd is not actually trustworthy. To put it another way: someone going and messing up their implementation of PartialOrd could introduce memory safety problems.

That is clearer. I would note though that the fact that unsafe may imply pointer math isn’t necessarily obvious. Most of the docs focus on if, let, destructuring, etc. It may not even be known that pointer math is an option. The most common usage I’ve seen of unsafe is transmute() I think but I have avoided them.

Send and Sync must be unsafe because if you implement them when you shouldn’t, you can end up violating memory safety somewhere else in the program.

I think I get this but it’s a bit unclear because I have no context or experience with either of the traits. I get the impression you could, via unsafe, send 2 references to a single item to different places and drop one under the guise of a valid Send trait and break the contract creating create memory unsafety. Or something like it.

Anyway, lack of experience with those traits may make those explanations puzzling.

---

Thanks. I’ll try to read further into it now.

[Gankro]

The key idea with Send and Sync is that there’s no way to check if something is threadsafe of “defensively program” around unthread-safe things (short of being single-threaded).

You can defensively program against a bad Ord implementation pretty easily by just checking things that “should” be true based on the properties of a total ordering (although there can be some minor perf losses in doing so for some algorithms).

However you can’t detect or guard against the fact that Rc actually contains shared mutable state that is manipulated without atomics (Rc looks identical to Box). Basically as soon as you send an Rc to another thread your program has data races (which violate memory safety, which safe code cannot be allowed to do). You need to have some fundamental notion of “is threadsafe”, and you just have to trust implementors to be correct. If they are incorrect, your program just isn’t memory safe. Since you have to trust them, and memory safety is at stake, the traits must be unsafe to implement since safe code can’t be allowed to violate memory safety.

[mdinger]

Okay, that makes sense.

[mdinger]

All other traits that declare any kind of contract really can’t be trusted to adhere to their contract when memory-safety is at stake. For instance Rust has PartialOrd and Ord to differentiate between types which can “just” be compared and those that implement a total ordering. [They could be implemented in 2 ways:

• unsafe: This would imply that when you unsafely rely on the unsafe implementation, an error such as failing to provide a Total Order might cause you to index out of bounds (for example via incorrect pointer math).
• safe: the same error which might cause violate memory safety before will not now because the tools to abuse mistakes (like pointer math) are not available.

]

However you can’t actually trust an implementor of Ord to actually provide a total ordering if failing to do so causes you to e.g. index out of bounds. But if it just makes your program do a stupid thing, then it’s “fine” to rely on Ord.

[This is the reason]The reason this is the case is that Ord is safe to implement, and it should be impossible for bad safe code to violate memory safety. Rust has traditionally avoided making traits unsafe because it makes unsafe pervasive in the language, which is not desirable. The only reason Send and Sync are unsafe is because thread safety is a sort of fundamental thing that a program can’t really guard against locally (even by-value message passing still requires a notion Send).

I tried to rephrase the part I didn’t think was clear. It may or may not be fully correct but it seems more obvious to me. You seem to be implying switching between unsafe and safe functionality without indicating so which would seem better as explicit.

Note: I have a decided preference against e.g. and i.e. because I always mix them up which is why I wrote “for example”. You obviously don’t so…

[Gankro]

@mdinger I’ll try to factor in your recommendations in a bit.

In the meantime, I got rustbook working, so now the book is hosted at http://cglab.ca/~abeinges/blah/turpl/_book/

[td_]

From the introduction:

On trait declarations, unsafe is declaring that implementing the trait is an unsafe operation, as it has contracts that other unsafe code is free to trust blindly.

I’m pretty sure the bolded ‘unsafe’ is an incorrect qualifier. Send and Sync are contracts that both safe and unsafe Rust are free to trust blindly, no? Or is it qualified because the fundamental code that cares about these contracts (e.g. the code in the std::thread module) is unsafe code?

[Gankro]

Yeah the key point is that unsafe code normally needs to be paranoid about bad safe code. It can “trust” other unsafe code though.

[mdinger]

@Gankro I hope you didn’t stall it because I didn’t respond…I didn’t want to just add a +1 so I didn’t respond. I haven’t gotten back to the rest of the document to examine it yet though. Maybe I’ll get to it this week.

[gnzlbg]

I don’t understand what is going on in this example, what is y? (this is possibly because I haven’t used if let much). It probably wouldn’t hurt if non-Rust programmers were able to understand this too.

let mut x = B(2.0);
if let B(ref mut y) = x {
    *x = A(7);
    // OH NO! a u32 has been interpretted as an f64! Type-safety hole!
    // (this does not actually compile)
    println!("{}", y);
}

I can’t wait to read the section on the drop_flag, you got me hooked from beginning to end with TURPL

[Gankro]

y is a pattern binding to a subexpression of x. In this case it’s getting a mutable reference to the number inside the enum. I’m probably going to simplify this section to just use Box or something, based on several issues people have had with it.

[gnzlbg]

A bit of nitpicking is the use of some insider terminology without explanation like borrowck and dropck. Rustaceans understand these, but adding a small remark the first time they are used would make it easier for non-rustaceans and newbies to follow. For example something like

• borrowck a.k.a. the borrow checker, that is, the static analysis “tool” that ensures correct borrowing.
• dropck aka the drop checker, that is, the static analysis tool that ensure correctness for drop.

[gnzlbg]

The page on coversions is underlined from the middle till the end.