Whether it’s possible or not, this sounds like the rules to determine when the mutex is unlocked become extremely complicated. Someone reading the code is unlikely to be able to tell at what point it is dropped, which makes reviewing the code for correctness very hard.
Is there really any reason to want “magic” rules over just adding in the appropriate drops when you want to early-release?
@fuji-184 it seems to me you're try to change the semantics of existing code, which is a breaking change. You're then trying to fix specific cases where this change would be breaking, by creating very complex and ad-hoc solutions for them. My opinion is that this approach will only lead to suboptimal and incomplete solutions.
For example: you are focusing on the case where the contents of the Mutex are converted to a raw pointer, but have you considered the case where the Mutex is used to logically protect some data that is outside of it? How would the compiler ever detect that? This might not even require unsafe code, e.g. if you use atomics and the correctness of the algorithm depends on one particular operation never being executed concurrently with itself.
There are so many ways this can go wrong that trying to ad-hoc fix them one by one is a lost battle.
If you look at the code, the locks are acquired in the same order. The issue stems from a "late join", because the main thread isn't blocked immediately after spawning a new thread, it becomes very easy to trigger a deadlock due to scope based drops. Why not block the main thread right away? Because the goal is to allow the main thread to continue executing other tasks in parallel, only blocking it when it reaches the point where it needs to retrieve the result from the worker thread.
No, Rust has auto trait propagation, for instance, if one of the child components is !Send, the parent automatically becomes !Send. The ASAP_drop trait would work the same way, so the compiler simply need to query whether a given T is ASAP_drop or !ASAP_drop.
This isn't an issue, as you can simply look at the parameters. There’s no need to worry about the order of operations, unlike with scope based dropping, A and B are not interdependent; they are entirely independent and do not reference one another, so concerning yourself with the order is pointless. "ASAP dropping" ensures that the lock held by A is released the moment it is no longer in use, the same applies to B, and, unlike scope based drop, it is not vulnerable to the "hierarchy footguns".
If let's say, one of the fields in B points to field A, an "ASAP drop" won't drop it, because A is still being referenced by B, in other words, it’s still in use, it will drop the value once none is using A. If it’s a pointer, as I mentioned earlier, it becomes a "scope based drop.". And the MutexGuard doesn't drop the value, it calls the unlock method to unlock the lock, the dropping value happens later by the T's impl Drop not the MutexGuard's impl Drop.
No, there are significant performance benefits :
It is memory efficient, scaling at O(n) relative to the size of the memory being held waiting end of scope and the duration until the end of the scope is reached. This results in a low peak memory usage, exactly equal to proper manual memory management (like C, Zig), because, in manual management, memory is freed immediately when it is no longer in use, rather than waiting for the execution to reach the end of a scope (which could involve many lines of code with unpredictable execution durations).
It offers low latency and high throughput multithreaded code, scaling at O(n) relative to the total duration of unnecessary blocking. To visualize this: if a channel processes tasks at 100 ns per task, a 200 ms thread block would result in the loss of 2 million tasks. This is a dynamic "footgun", it isn't fixed, the longer the blocking duration, the greater the loss in latency and throughput.
No, that’s easy, because it’s exactly the same as tracing which function last passed A as a parameter, or which variable last pointed to A,it’s all clearly written in the code. For example :
What are alreasy exists :
let a = A { }
fn_name(&a)
let b = B { a: &a }
let locked = mutex_guard.lock().unwrap()
locked.write(20)
locked.write(50)
It seems you didn't read my opening, because I clearly mentioned this :
There is no change in the outcome, the results remain the same, but it offers better performance, scales at O(n), and is more resilient against deadlocks. Remember, complexity can not simply be eliminated, if you make things "simple" at the compiler level, that complexity just shifts onto the user, manifesting as performance pitfalls and the deadlocks I mentioned earlier. Conversely, if the compiler absorbs that complexity, the result is more optimal for the user right out of the box. In all my design work, I’ve never encountered a case where the final output differed, so let’s avoid a "this is the only way" mindset. Try to adopt a more supportive tone, it encourages people to contribute and keeps them enthusiastic about experimenting. Your discovery of that special case is excellent, and I am currently considering a solution.
Sorry but simply stating something won't convince me it's true.
The solution you are proposing is not complex for the compiler, but rather it has complex semantics. That is, it will be complex for me as a user to undestand when a MutexGuard will be dropped.
Let's also avoid a "I don't see an issue so there must be none" mindset too. The burden of the correctness proof is on who proposes a feature, not on who has to find where it breaks. That's why I told you that ad-hoc fixes are the wrong approach, they won't be convincing enough.
I try to be supportive when I see something that makes sense and seems doable, but I don't like giving false hopes to people when they're following an approach that I believe to be fundamentally flawed.
It's not just null pointers you have to care about, though - any pointer could be problematic, and I could reimplement my example without use of null pointers via allowing use-after-free instead (also UB).
But if you say that ASAP drop cannot apply if any of the data structures protected by the MutexGuard contain a raw pointer, then ASAP drop does not apply if the contained thing is any of the standard collections, or a smart pointer (since all of those contain raw pointers of various forms). This then forces you down the route of marking everything that contains a raw pointer as "safe for ASAP drop", with something along the lines of unsafe impl<T: ASAP_drop_safe> ASAP_drop_safe for Vec<T> {}.
This starts to feel like an awful lot of complexity to allow you to swap writing drop(guard) if you want a drop before the end of the scope for writing drop(guard) to explicitly mark cases where you want the lock's duration to be extended.
Add in that profilers can and do tell you which locks are contended, which code paths hold the lock when it's contended, and how long they're contended for, and it begins to feel like a lot of work for minimal gain - anyone who actually cares about performance is going to profile their code, and if the difference between early drop and scope-based drop matters (and it's safe to do so - no funny business with raw pointers, or depending on a lock being held after the last access of the data it contains), they can trivially add drop(guard) to release the lock early, and it reads fairly naturally (whereas drop(guard) to extend its lifetime feels a bit weirder).
That said, none of this argues against a Clippy lint for this case - and indeed, clippy::significant_drop_tightening exists, but is in the "nursery" for now. Working on resolving the known issues with that lint so that it can be promoted would be a good use of your time and effort - and note that there's a path for lints that started in Clippy to get promoted all the way to deny-by-default rustc lints.
A demonstration of output interleaving (in a pipe, as that's more convenient to show). Rust Playground : lock_writer and lock_earlydrop_writer have identical code except the early drop, and yet have different behavior. The first is guaranteed to write [][][][][], while the latter will interleave brackets written from different threads. Note that it has no unsafe code either.
I stand by the opinion that programmers are capable of writing code that represents their intent, if code's behavior is as written.
In practice, i.e. code I saw and code I wrote, either the closing bracket is after data.update, or the mutex should've been held until later.
I stand by the opinion that moving Clippy - lint-by-lint - into rustc is better achieved in other ways, like advising to run the aforementioned Clippy as it is now.
This is not generally known until after monomorphization. If anything the like will be accepted, I think it’s rather a dedicated mutex type that behaves like this and has a bound on ASAP_drop.
When would this check happen anyway? It would be very awkward if the mutex lifetime would not only depend on type internals, but also on how much about it is known (like, you may have an ASAP_drop type behind a projection that can’t be normalized away pre-monomorphization). Even if it happened after monomorphization, it would still be very awkward that in different instantiations of generic functions, such a mutex would have different lifetimes depending on which types the function is instantiated with.
There is always a solution if try creative, if don't want to break specific legacy code cases that have already been written. Rust could simply introduce a new lock type, ASAP_Mutex. This way, the Rust language wouldn't be forever shackled to the "sins of the past", namely, the performance pitfalls and deadlocks that frequently arise from scope based drop behavior.
I said that prior to that specific case, there was no difference in the outcome.
No, I already said it's very easy because it's equivalent to knowing who the last user is, it is exactly equal to "who the last user of &mut".
And because how ASAP drop works automatically guarantee isafe from performance "footguns", and, as an added bonus, it is also more resistant to deadlocks, so users don’t have to go through the trouble of reviewing code as much as they would with scope based drop mechanisms, reducing the need of manual review.
Not having found something yet is not the same as saying it doesn't exist, please don't put words in my mouth. It was clear that I was inviting others to conduct a review, specifically to catch things that I might have missed. I clearly said this :
And no, there is no liability, this is open source. The whole point of open source is for everyone to contribute. If the task of identifying flaws were left solely to the person who started the thread, then, quite frankly, there would be no purpose in opening the discussion at all. The value of a discussion lies in mutual contribution, for instance, if I see an idea with great potential but notice a shortcoming, I will point it out, and if I happen to know a solution, I will share that as well. From there, others can follow up, potentially leading to even better solutions. It’s like having ten holes that many people work together to patch until they are all completely sealed.
So, are you just killing people's enthusiasm based on your own personal opinion? I’ll share my experience discussing things on the Zig forum, perhaps it will give people something to reflect on. Over there, the member is incredibly supportive, which makes the whole process enjoyable. Here, whenever you put forward a proposal, people just criticize it without offering any solutions or helpful information. Criticizing without providing a solution is actually ok enough there is real contribution, but the real issue, and I’ll be honest here, is that the tone people use in this space just kills one's enthusiasm.
No, only raw pointers are incompatible, because they are the only ones that lack lifetime information. Smart pointers are compatible with ASAP drop, because the original value will not be dropped automatically as long as any cloned instance remains alive. From the compiler's perspective, these clones refer directly to the original value, rather than to one another. This is actually advantageous, as it prevents errors such as "the original value was dropped prematurely because it went out of scope."
There is a way to use it explicitly, which I mentioned in a previous post about ASAP drop, to tell the compiler: "Trust me, the ASAP drop here is guaranteed to be safe." For example :
let a = unsafe { asap String { ... } }
In practice, you can simply use it within the constructors of collections found in the standard library.
What is the issue with updating standard library code? Isn't progress defined by moving forward rather than staying in one place? Consider some major advancements previously made in Rust :
updating the standard library to support custom allocators
adding try APIs so that allocation failures return a Result instead of causing a crash.
No, actually, writing manual `drop` calls is poor practice, as it creates equal "footgun" scenario to manually calling free, no better than it. In ASAP drop, user don’t need to do that, user simply read from or write to the variable, the lifetime of the lock is automatically extended, it lasts exactly as long as it’s needed and is cleaned up immediately afterward, with no risk of lifetime leaks that degrade multithreading performance.
I’m sorry to have to say this, but you seem to be coming from a conservative rather than an innovative mindset, please try to move past that, the issue here is simply that the tools don’t yet support new Rust language features, which shouldn’t be used as a justification to block the adoption of those features. Do you honestly think that when Rust introduced the custom allocator API, every single library suddenly and magically became compatible with it? Of course not. Profiling tools get updated to support new language features as they are introduced, you’re overreacting.
Furthermore, your claim that writing drop manually is "ergonomic" is fundamentally incorrect, it’s effectively the same as manually calling free. The reality is that existing codebases don’t do this, so your idea is purely wishful thinking, the fact that it isn’t being used proves it’s not ergonomic. It also suggests that many people may be inadvertently falling into a "performance footgun" in Rust without even realizing it.
Can this lint distinguish raw pointers? Furthermore, does it allow usage on raw pointers that have been manually verified? If it can handle the scenario mentioned above, and allowing for a "deny by default" in rustc level, and specifying exactly which line requires a drop call, then it would indeed yield the same result. In that case, I agree, as long as the outcome is identical, nothing is lost.
The only difference is the method: with a compiler, everything happens under the hood, the user has smooth experience, whereas with a linter, the user will see a multitude of errors and must manually write the fixes for them. But for me personally, it’s acceptable.
Edit :
Oh no, since the ASAP drop isn't just for releasing locks but also for memory management by default for code that uses reference or annotated with keyword asap explicitely (for raw pointer that is proven safe), relying on clippy lint that is adopted to rustc's default error would make the developer experience quite grueling. Rust users generally lack experience with manual memory management, they will write code using Vec, String, etc., only to hit compiler errors, forcing them to manually call drop, repeating this cycle over and over. This significantly lengthens the development iteration loop, which, compounded by slow compile times, becomes a major pain point, not to mention that the code ends up being overly verbose. With my approach, none of this happens, because the compiler absorbs that complexity and handles it automatically. Personally, I’m fine with it, but I’m not sure how the average user would feel.
No, you haven't demonstrated the use of drop for early release of locks, and your scope-based locking approach is highly fragile, tt’s prone to the performance "footguns" currently being discussed if there is more code following it. You write like this :
But the actual code that users will end up writing looks like this.
fn lock_earlydrop_writer(mut w: Mutex<&PipeWriter>) {
let guard = w.lock().unwrap();
guard.write_all(b"[").unwrap();
guard.write_all(b"]").unwrap(); // mutex is guaranted unlocked after this line even if there are many following code
// many others code
}
Or if you want to use that global variable approach, Mutex as token not guarding data directly :
fn lock_earlydrop_writer(mut w: &PipeWriter) {
let guard = w.lock();
guard.write_all(b"[").unwrap();
guard.write_all(b"]").unwrap();
let _ = guard
}
My earlier code is much better because it removes completely the possibility of footgun the mutex lifetime being extended unnecessarily.
No, this is just wishful thinking, not reality. If the majority of programmers were actually capable of handling it, the borrow checker could simply be removed. In fact, many Rust users have a limited deep technical understanding of memory management, precisely because Rust doesn't expose manual allocation, they never learn how to manage memory manually. As a result, they end up performing unnecessary heap allocations that they don't know it calls malloc, most commonly by creating unnecerery String objects. Honestly, your comment isn't based on the reality on the ground, Rust users almost never call drop manually, they write "vanilla" Rust code (code that requires no manual effort).
No, in real world public Rust code, people don’t write explicit scopes for locks, they just keep them within the same scope (this is standard, idiomatic Rust, how most users actually write it). Doing otherwise would require manual effort, comparable to the knowledge needed to manually manage memory (like calling free). In real world public Rust code, people simply don’t worry about that.
See? Your method is based on the wishful thinking that all users will follow the rituals you promote. In reality, they don’t, because the "footguns" I mentioned earlier exist in their code. You need to design a method based on how things actually play out in the real world, otherwise, it’s just a waste.
It’s not solely opinion; it’s founded on knowledge of how the Rust language and compiler work. We’ve seen enough weird Rust code that we can think of edge cases that make various ideas very difficult to implement.
I truly think that the clippy lint would be a good solution! Your argument was that not everyone uses clippy, and mine would be that more people should use clippy. Plenty of downright bad code is caught by clippy. (Plus, not everyone would use AsapDropMutexGuard.)
I don’t know how well it translates, but AFAICT most responses to you are written in good faith. I have a lot of fun learning about Rust on these forums, and sharing interesting information with others around me.
I wonder if the issue is “problems in search of a solution” vs “solutions in search of a problem”… if your goal is to solve certain problems (like performance issues with mutexes), I think you might have a better time on the forums if you presented problems you’re having, and maybe a half-baked thought on how they might be solved. Like this thread, for example. And then of course people will argue that certain solutions (possibly the one suggested by the poster of the thread) are bad, because the goal is to solve a certain problem, not “this solution is really cool, I want to try using this solution”.
Conversely, if your goal is “early-dropping of types is a really interesting idea, I think it would be cool if Rust supported early drops, and I want to find some benefits of this feature. It seems useful for mutexes”, then the responses here probably would’ve focused on finding possible use cases for early drops, rather than on solutions you don’t care about.
(Though there would still be concerns about how to implement early drops… I lean towards thinking that early drops should be purely for performance optimization, not for correctness, and that the exact time a value of an early-droppable type is dropped should be somewhat nondeterministic; it should just be guaranteed to be dropped somewhere after the last usage of that type and not after the end of the drop scope of that value, such that not early-dropping early-droppable values would be a valid implementation of Rust, and the extent to which they are dropped early would be a quality-of-implementation concern.)
Having a dedicated ASAP_Mutex type is also acceptable, I mentioned that earlier, too. It’s the correct approach because it won’t break existing code that already uses a mutex as a token, nor does it lock the Rust language into the shortcomings of the past indefinitely, new Rust code can begin to reap the benefits of ASAP drop.
No, saying that there’s seemingly no way forward is just an opinion. I already answered the issue mentioned can simply be resolved by creating a dedicated ASAP_Mutex type, ensuring that legacy code doesn’t hold back a better future.
The fact is, even after years of Clippy being in use, the "footguns" I mentioned earlier are still very prevalent, which means that this particular approach has failed to completely prevent them. While ASAP_Mutex would be explicit, the benefits it provides for memory management are gained automatically, without requiring any manual effort. Once Rust developers experience the advantages of ASAP_Mutex, and given that it’s a component of the standard library rather than an external tool, it will naturally become the go to mutex, replacing the legacy one in new Rust code.
It’s simple: I have an idea with promising benefits. It can be automated within the compiler to ensure a seamless user experience. I’ve already identified several advantages (A, B, and C) and addressed every drawback I encountered by refining the design. If you spot any new weaknesses, please note them, if you know of a solution, share that as well. Even if you only have partial information that might help, even if it’s not a perfect fix, please write it down. By sharing your thoughts, you might enable others to build upon them or inspire an even better solution. Let’s keep the tone supportive to maintain the enthusiasm.
To summarize, the latest design for "ASAP drop" is as follows :
Variables referenced by pointers do not automatically use ASAP drop.
All other variables automatically use ASAP drop.
ASAP drop will have its own implementation trait, for example: impl ASAP_Drop for T { fn drop... }.
During code generation, the compiler checks for an ASAP_Drop implementation, if found, it analyzes the variable's last usage and invokes the drop function defined in that implementation. If no ASAP_Drop implementation exists, it defaults to scope based drop. If both are present, ASAP_Drop takes precedence.
Using ASAP drop for pointers and the variables they point to must be explicitly enabled and verified, this is done by using the asap keyword (which can be applied either to the type or at the point of variable declaration).
All existing locks in the standard library will continue to use scope based drop.
A dedicated set of lock types will be created for the standard library that utilizes ASAP drop.
The dedicated ASAP lock types does not drop the value it holds, it merely calls the unlock method.
Dropping the value contained within an ASAP lock<T> type is the responsibility of the ASAP_Drop implementation belonging to T itself.
I'd encourage you to try and write user-facing (!) documentation as to when ASAP_Mutex can and cannot be used, when it is and isn't preferable over the standard Mutex, and what the corner cases are where the user needs to address the different ASAP_Mutex behavior, and how.
And no, "one can simply replace Mutex with ASAP_Mutex in every case and things work every time" would be incorrect and insufficient, as @ProgramCrafter's Rust Playground counter-example still stands (and yes, INB4: I did read your answer to that).
There's 2 cases of pointers:
pointers pointing outwards from the guarded data structure: that seems doable, although as mention it applies to most std containers
pointers pointing inwards (in)to the guarded data structure: it's still unclear to me how that can be reliably detected by the compiler, in order to prevent a use-after-free due to an ASAP_Mutex potentially dropping stuff earlier than anticipated.
Every drawback being addressed at least once doesn’t entail that every drawback has been sufficiently addressed.
To emphasize one part of my comment: I think the way to make it implementable is to literally make the exact time an early-droppable type is dropped not be preserved by compiler optimizations, be permitted to change in non-breaking updates to the compiler, and so on.
(Why? Because the time that a value “stops being used” can depend on when it “stops being borrowed”. Currently, the borrow checker is a pass/fail test that does not affect the behavior of any code that compiles, in part because we want to reserve the ability to improve the borrow checker and/or trait solver. Such improvements could change when variables are considered to no longer be borrowed, which would change when an early-droppable type is dropped.)
Which is a nontrivial addition to the language, though there is precedent in the global allocator (memory allocations can be optimized out).
The relevant clippy lint is a nursery lint, meaning that it hasn’t been fully implemented, and (because almost nobody enables nursery lints) almost nobody is using this lint. Thus why “fully implement that lint” would be a good improvement.
Practically speaking, “replace all” is a breaking change. Is it legally speaking? The documentation says that Mutex protects its payload, which it does even in this example. With the new semantics, no data is protected for the entire duration of its usage (none). With the old semantics, it protects no data until the end of the scope. I understand that it’s tempting to piggyback on the payload-independent locking behaviour, but per documentation, is this a valid use case?
Let’s assume we had a weird magic OS locking primitive that is based on address ranges. Basically, blocking doesn’t happen when acquiring the lock, but only when accessing the locked memory (maybe the memory is only mapped when the lock is held). Once the (basically deferred) lock request succeeds, memory accesses are unblocked. Now, anything between locking and the first access to the protected data can still freely run concurrently. Would this be a valid primitive to implement Mutex with? I think my main issue is that the documentation clarifies the duration of locking, but is not crystal clear on the (side) effects of locking.
Off-topic ramble: drop has been compared to free, which ergonomically speaking is somewhat true. However, most of the time, when an object is freed does not really matter. Considering critical sections should basically always be best-effort minimal, locks are a special case to me. RAII really doesn’t sit right with me, because drop is optional and doesn’t directly correspond to lock. The lock’s scope can range from the middle of a block to its end, which is not very nice to model mentally. I feel like my personal preference is that either there are traditional lock and unlock operations (where implicit drops would yield a build error), where the scope is actively bounded in both directions the same way, or something like a with_lock method that invokes a closure, with the scope implicitly bounded by the closure’s. I know that closures can introduce their own limitations and ergonomic issues, but this is a ramble after all…
I have a case where I didn't want the lock held for the entire scope. However, even then, the solution is not early drop because the issue caused by holding the lock for too long was solved by using existing Rust facilities to make sure it never happens again rather than "getting lucky" with the ASAP drop mechanism that could have allowed the bad pattern to be written again in the future.
See this old post, but the changes are in this MR. This commit is the "ASAP drop" fix. but this followup commit is the "never allow it to happen again" fix where the compiler ensures that it never happens again. While ASAP drop would have avoided this bug in some of the methods (member borrowing would likely have extended the lock in various cases; Copy in others avoiding the bug), I think I'm happier with the compiler-enforced mechanism that now exists instead.