Seems like an annoying environment… personally, I’d blame lack of company-provided training on unsafe before blaming individual devs.
And counter point, there are crates out there which are TRIVIALLY extremely unsound. Like, “I open it in docs.rs, look at the source of a function, and see undefined behavior within a few seconds.” And you can bet that people who don’t know how to write unsafe also aren’t capable of noticing those problems so quickly if at all.
It is safe as Rust defines it.
Maybe I'm not understanding. As far as I can tell if two threads mutate the same file description through separate descriptors that's effectively the same as two threads directly modifying the description and even though the kernel ensures that this is safe from its perspective race conditions can still arise from things like SOCK_STREAM. Is that not covered by memory safety?
I never said upper management was rational 
And yeah the access controls made doing your job annoyingly difficult.
Yet OwnedFd and BorrowedFd cant safely encapsulate full functionality, such as fd inheritance...
The kernel file descriptor implementations are thread-safe. E.g. the seek-position is more like an AtomicUsize, concurrent writes don't clobber each other (unless they're positioned writes), thought they might get interleaved in some piecewise manner. Otherwise multiple threads writing to stdout (be it a file, pipe or tty) wouldn't work.
This does not cause memory unsafety or undefined behavior. It might cause data corruption from an application logic perspective, but not from filesystem data integrity perspective.
There are file descriptors where incorrect use can lead to undefined behavior, but they're things like device descriptors (talking to some hardware), epoll FDs (messing with the user pointers likely causes delayed UB), memory-mapped files.
In those cases you need exclusive ownership of those fds and shouldn't hand out borrows unless those borrows are also properly synchronized in userspace. But those things shouldn't be opened as a File, they shouldn't implement AsFd and so on (AsRawFd would still be fine, as doing anything with such a RawFd should be marked unsafe and it's up to the caller to use it correctly).
If the wayland protocol used SOCK_SEQPACKET instead of SOCK_STREAM it might even be reasonable to process messages on multiple threads in an unsynchronized manner. But it doesn't, so yeah such an FD should have a single owner (that owner may be a lock allowing multi-threaded but serialized access).
Not a company I would ever consider working for. All places I have worked for practised blameless culture. If something serious happens that is a time to reflect on what was wrong on a systemic level and what can be done to prevent such errors in the future.
It might cause data corruption from an application logic perspective, but not from filesystem data integrity perspective
I think this might just be a terminology difference then (or i miss wrote my comment), my understanding of program safety is that the end goal is to eliminate undefined behavior and corrupted data is the main symptom of that and the justification for developing a "safe" systems programming language in the first place.
If two threads can corrupt data output to a file descriptor without being stopped by the compiler by not being careful is that not a safety violation?
Devs at these companies typically worked in c#. In my time at those companies I never once saw code with an unsafe block. The idea that people who have likely never dealt with unsafe code in their life wouldnt take issue with that is kind of nuts. And yes unsafe blocks being a requirement in user code would definitely cause issues. Especially for something as simple as fd inheritance.
Not quite. The thing about memory safety is that if it is violated, any data in the process could be corrupted, whether or not it is related to the data your code intended to work on. That potential action-at-a-distance is what Rust tries to eliminate, but it is not the same thing as corrupting data that the code is working on. That can’t be eliminated by static checks, because the difference between correct data and incorrect data is application-specific.
Similarly, Rust’s “IO safety” is designed to prevent accidental access to the wrong file descriptor, which could corrupt a file or other resource unrelated to what was being attempted, but it has no opinions on how a certain file descriptor should be used.
It does seem pretty niche to me. But yes it would be nice if it didn't require unsafe. However, unsettling the environment variable will always require unsafe on POSIX systems. So if that is part of your requirement, you won't be able to get rid of unsafe.
I have worked in C++ (and later Rust), embedded and industrial. Yes, doing unsafe (or breaking coding guides in C++) would certainly require a good documented motivation and double checking by colleagues. But every non-trivial project will have some of that.
Rust is a systems language. I would expect anyone with C# background have to learn about systems programming regardless of if they go to C, C++, Zig or Rust. Just like anyone writing low level concurrent code will need to learn about atomic memory orderings.
That sounds like making an ownership guard for a reference when everyone can bypass the guard, get a copy of the reference, then start mutating the referenced resource at will. What's the point of preventing accidental access to the wrong file descriptor when two file descriptors could point to the same description and mutations with one would mutate the other?
If a file descriptor is duplicated, some code did the duplication on purpose. If IO safety is maintained, you know that if a file descriptor isn't intentionally duplicated, there won’t be duplicates caused by bugs elsewhere in the program.
Certainly it would be better if there was a way to coordinate with the kernel to have known exclusive access to a resource. But there just isn’t a system API with which to achieve that.
Just because you have the ability to access a resource does not mean you should.
It does not aim to prevent intentional calls to try_clone on resources you properly have obtained ownership of.
If you have a resource where try_clone would be inappropriate then move that OwnedFd into an opaque wrapper as soon as you have obtained it and then nothing which obeys IO-safety will touch that FD. See Does rust abstract file operations incorrectly? - #5 by the8472 for how to do that on inherited FDs.
I would note a couple of things:
As an example of the first point, take this safe code with a data corruption bug - it's a dumb example, but because I got the locking badly wrong by doing the "obvious" thing (and then put some sleeps in to make it very likely that I'd see the corruption), I lost a string that was supposed to be printed. However, while this program is technically non-deterministic (although the sleeps make it very unlikely that you'll see different behaviour to me), it's fully defined behaviour - it's just that print_slowly_then_clear releases locks at times when you might not expect it to from the limited documentation.
For the second, this is what SAFETY comments are about when it comes to the controlled use of unsafe - because Rust (absent soundness bugs) constrains UB to only occur within unsafe blocks, you know that your program's behaviour is fully defined (if surprising and annoying) if you can confirm that the unsafe blocks don't do anything that executes UB. The SAFETY comments provide a record of people's thinking on this topic, so that you can be confident that bugs you're seeing aren't the result of UB, but are the result of badly thought through program logic.
And the reason that UB is so feared is not that it results in data corruption - lots of bugs result in data corruption - but rather that (via the Curry-Howard correspondence) a program with UB is equivalent to a logic proof of a contradiction (e.g. 1 == 2), and a proof of a contradiction can be to show any proof is correct. In turn, going back via Curry-Howard, that means that any program that contains UB is equivalent to any other program - and that is why UB is feared.
You have so far not given any argument for how the different treatment of what is and isn't safe would help with FD inheritance. You are just claiming it will magically help somehow? Please tell us how. Sketch the APIs. Describe their safety requirements. Tell us why that still solves the problems I/O safety solves, while also supporting FD inheritance.
Yeah, so what? Just like the borrow checker cannot encapsulate everything you can do with memory. So, no surprise here. No safety system is sound and complete; I prefer the sound and incomplete ones over complete and unsound ones. That's why we have unsafe code, to ensure that incompleteness does not rule out valid usecases.
Indeed. And you cannot cause Undefined Behavior just by calling FD operations concurrently on the same underlying file description. That's why in general we don't have to isolate and track file descriptions the way we have to isolate and track memory -- file descriptions are fundamentally thread-safe objects, in the same sense AtomicUsize is: if you get the concurrent access logic wrong you may get surprising results, but the program cannot catch fire just because of it. (Once there's also unsafe code involved the game changes of course.)
Uh... no, I cannot think of any way in which that is correct.^^ You cannot even meaningfully apply the Curry-Howard correspondence to Rust since Rust allows infinite loops. Infinite loops are proofs of every logical statement and therefore of contradictions. Programs with UB aren't valid programs and therefore not a proof of anything, not even a contradiction.
UB is a violation of the contract between programmer and compiler. See this blog post for a longer explanation. But it has nothing to do with Curry-Howard. A program with UB is also not equivalent to any other program (by transitivity that would make all programs equivalent with each other), it is just refined by every other program.
Also, everyone, please keep the discussion of company culture out of this thread, it doesn't contribute to the topic.
My concern now is partly with how the language portrays io operations as safe (mainly at issue is when two threads using entirely safe code (try_clone) could operate on the same file description cursor to get a race condition from output order. I guess technically this is not a memory safety problem per say but it still seems like something that rust could fix) and how to do fd inheritance safely as a separate problem. My questions sort of evolved into asking about both over time so sorry if that was confusing.
As to the problem of making fd inheritance safe I've had some time to think about it and it seems (for posix compliant systems anyway) since any new fd a process gets is required to be the lowest unused fd available then all the rust functions that interop with the file system (think dup, open, creat, accept, etc.) could update an atomic MAX_FD c_int-typed value with the largest that has been seen (read "returned") so far. As long as there's some mechanism keeping track of jumps of greater than 1 of that number (probably just a Vec<c_int> or something) then a safe function could be created that would allow user code to "claim" a c_int as a valid "primordial" fd as long as it's
fcntl or something) andVec<c_int> (the fd would need to be removed once handed out) ORdup and close the fd given (remembering to update MAX_FD) to ensure that inherited fd's far past MAX_FD don't cause a range of unclaimed fd's (which may or may not be open primordials) to be excluded from other claimants while also preventing future claims of that fd)Feel free to critique, but that seems like a fairly robust system for maintaining the single ownership requirement while shoehorning in the edge case of inherited fd's
While it sucks that this can't be made as a standalone crate and would require callbacks to this MAX_FD tracking routine to be inserted in between any fd creation functions at least it shouldn't cause breaking api changes
In general, Rust is not trying to rule out code doing things wrong, since that is impossible.
You can get a race condition by (safely) misusing atomics or mutexes, or — directly analogous to a file descriptor — cloning a MPSC channel Sender and having two threads send messages such that the ordering between them makes no sense.
But all of these can be minimized by keeping the atomics, mutexes, or channels under the ownership of code that handles them correctly — encapsulation. That’s the other thing I think your analysis is missing: under the IO safety system, the way to prevent a file descriptor you own from being cloned and misused is to not hand it out. Keep it in a private field. Maintain control.
Are you also concerned about AtomicUsize operations being safe? That also allows race conditions in safe code. This is fully intentional.
If you want to define an unclonable FD type where you expose the FD to users but don't let them call dup on it, you can do that. But the usecase is not considered common enough to warrant inclusion in the standard library.
unless you can e.g. modify the C library and the Go compiler (because Go likes to do raw syscalls and bypass the C library) to insert that code in there, this seems unworkable because as soon as you call some non-Rust library, it loses track of which FDs were created, since that code has no obligation to call your MAX_FD tracking functions, and on Linux code can even do system calls without calling out to any library (since that is stable on Linux, unlike almost any other OS where you have to call the libc wrapper functions).
e.g. here's a C++ program that doesn't use any C/C++ library functions at all and instead does syscalls using inline assembly to open a temporary file, write a message, close it, open it again, read the contents, close the file, and finally write the read data to stdout:
AFAIK calling non rust code would have to be done through ffi bindings of some sort which is where I’d imagine the book keeping for this would go. I’m not 100% on that as I have very little experience making ffi bindings but if someone’s code is performing unsafe stuff like calling c++ code the onus would be on them to ensure it works with the safety system. Although it’s not like this conflicts with the maxfd system, if the ffi code gets a raw fd there’s only two options, either it’s <= max fd in which case no one can “safely” claim it anyway OR it’s > max_fd in which case you would just push maxfd up to it potentially adding the jumped fds to the jump list and calling it a day.