# What is the current safety story for library-based stackful coroutines? **URL:** \ **Category:** Unsafe Code Guidelines\ **Created:** [June 10, 2021, 1:58pm UTC](https://internals.rust-lang.org/t/what-is-the-current-safety-story-for-library-based-stackful-coroutines/14863 "2021-06-10T13:58:26Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![matklad](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/matklad/32/12266_2.png) [@matklad](https://internals.rust-lang.org/u/matklad)\ **Post date:** [June 10, 2021, 1:58pm UTC](https://internals.rust-lang.org/t/what-is-the-current-safety-story-for-library-based-stackful-coroutines/14863/1 "2021-06-10T13:58:26Z")
One interesting reslult of the last round of tech empower benchmarks is that one of the top slots is occupied by may-minihttp: > **[TechEmpower/FrameworkBenchmarks](https://github.com/TechEmpower/FrameworkBenchmarks/tree/a77eb84565d4d5f494c67296c80474455130b329/frameworks/Rust/may-minihttp)** > > a77eb84565d4d5f494c67296c80474455130b329/frameworks/Rust/may-minihttp may is a **stackful** coroutine library -- it uses [asm tricks](https://github.com/Xudong-Huang/generator-rs/blob/master/src/detail/x86_64_unix.rs) to implement cooperative scheduling of green threads. However, may is not safe to use -- spawning a coroutine is an `unsafe` operation, because acessing TLS from a couroutine can lead to undefined behavior (and not just errors): [may/coroutine\_impl.rs at 5e1fe6392360267c092e7c5879ab96e5d68fc28d · Xudong-Huang/may · GitHub](https://github.com/Xudong-Huang/may/blob/5e1fe6392360267c092e7c5879ab96e5d68fc28d/src/coroutine_impl.rs#L322-L326). I am wondering, do we have some kind of a story to make this safe? Implementing green-threads "in userland" by manually switching stacks is a well-known systems programming trick. Can Rust express it in a sound way? ---
**Author:** ![mcy](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/mcy/32/6512_2.png) [@mcy](https://internals.rust-lang.org/u/mcy)\ **Post date:** [June 10, 2021, 2:30pm UTC](https://internals.rust-lang.org/t/what-is-the-current-safety-story-for-library-based-stackful-coroutines/14863/2 "2021-06-10T14:30:50Z")
Is the _only_ problem that TLS will be shared between coroutines that happen to be scheduled on the same thread? I wonder if it's sufficient to just add an intrinsic like `LocalKey::disable()` that causes all `LocalKey::with()` to panic in the current thread (and maybe a `with_unchecked()` for unsafe access). This seems like it would just cost an extra TLS variable that gets checked on TLS access, and an extra `unlikely` branch. Of course, if you touch TLS outside of Rust you lose, which no one expects to be a problem they have to deal with. ---
**Author:** ![bjorn3](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/bjorn3/32/2736_2.png) [@bjorn3](https://internals.rust-lang.org/u/bjorn3)\ **Post date:** [June 10, 2021, 3:18pm UTC](https://internals.rust-lang.org/t/what-is-the-current-safety-story-for-library-based-stackful-coroutines/14863/3 "2021-06-10T15:18:15Z")
> [@mcy](#): > > This seems like it would just cost an extra TLS variable that gets checked on TLS access, and an extra `unlikely` branch. This would be twice as expensive as the already relatively expensive single TLS access. ---
**Author:** ![mcy](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/mcy/32/6512_2.png) [@mcy](https://internals.rust-lang.org/u/mcy)\ **Post date:** [June 10, 2021, 3:26pm UTC](https://internals.rust-lang.org/t/what-is-the-current-safety-story-for-library-based-stackful-coroutines/14863/4 "2021-06-10T15:26:30Z")
I mean, I suppose you could do something like hang a reference into that TLS off of every other `LocalKey`'s TLS contents, but that also seems expensive in terms of memory. I can't think of a way to do this that doesn't incur some nasty cost even if you never call this function. (Oh, and this version is potentially self-racy so I think you lose no matter what. =D) ---
**Author:** ![josh](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/josh/32/5934_2.png) [@josh](https://internals.rust-lang.org/u/josh)\ **Post date:** [June 10, 2021, 3:30pm UTC](https://internals.rust-lang.org/t/what-is-the-current-safety-story-for-library-based-stackful-coroutines/14863/5 "2021-06-10T15:30:14Z")
One option would be to actually allocate TLS for each coroutine, and set up the appropriate register when switching. ---
**Author:** ![bjorn3](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/bjorn3/32/2736_2.png) [@bjorn3](https://internals.rust-lang.org/u/bjorn3)\ **Post date:** [June 10, 2021, 4:06pm UTC](https://internals.rust-lang.org/t/what-is-the-current-safety-story-for-library-based-stackful-coroutines/14863/6 "2021-06-10T16:06:28Z")
That would require intimate knowledge about the libc and libpthreads implementation. A quick search for how to set the TLS manually gave the following comment about it being hard: [It's hardly well known, but I implemented an SMTP greylisting proxy called Spey ... | Hacker News](https://news.ycombinator.com/item?id=9404034) ---
**Author:** ![josh](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/josh/32/5934_2.png) [@josh](https://internals.rust-lang.org/u/josh)\ **Post date:** [June 10, 2021, 7:33pm UTC](https://internals.rust-lang.org/t/what-is-the-current-safety-story-for-library-based-stackful-coroutines/14863/7 "2021-06-10T19:33:47Z")
@bjorn3 While it's absolutely _difficult_, it should possible to do in a way that cooperates with other libraries in the same address space. There are conventions to follow for how TLS gets allocated, and there's other code out there that manually allocates TLS. ---
**Author:** ![Soni](https://avatars.discourse-cdn.com/v4/letter/s/a3d4f5/32.png) [@Soni](https://internals.rust-lang.org/u/Soni)\ **Post date:** [June 10, 2021, 11:12pm UTC](https://internals.rust-lang.org/t/what-is-the-current-safety-story-for-library-based-stackful-coroutines/14863/8 "2021-06-10T23:12:31Z")
> [@mcy](#): > > Is the _only_ problem that TLS will be shared between coroutines that happen to be scheduled on the same thread? Wait, what's the issue of sharing TLS with things running on the same thread? Isn't that just how threads work? Wait are these coroutines Send or something? ---
**Author:** ![josh](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/josh/32/5934_2.png) [@josh](https://internals.rust-lang.org/u/josh)\ **Post date:** [June 10, 2021, 11:16pm UTC](https://internals.rust-lang.org/t/what-is-the-current-safety-story-for-library-based-stackful-coroutines/14863/9 "2021-06-10T23:16:31Z")
> [@Soni](#): > > Wait, what's the issue of sharing TLS with things running on the same thread? Isn't that just how threads work? If you're swapping back and forth between different userspace coroutines on the same thread, and the TLS is associated with the thread, you could have two different coroutines access the same TLS in ways that violate borrow-checking rules; for instance, you could borrow something as `&mut` from TLS, swap coroutines, and then the other coroutine could do its own `&mut` borrow of the same TLS without knowing about the first borrow. It might be possible to avoid that by using async functions as the coroutines, and only allowing switches between coroutines when awaited; however, that may or may not fit the desired coroutine model. ---
**Author:** ![bjorn3](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/bjorn3/32/2736_2.png) [@bjorn3](https://internals.rust-lang.org/u/bjorn3)\ **Post date:** [June 11, 2021, 10:08am UTC](https://internals.rust-lang.org/t/what-is-the-current-safety-story-for-library-based-stackful-coroutines/14863/10 "2021-06-11T10:08:27Z")
> [@josh](#): > > If you're swapping back and forth between different userspace coroutines on the same thread, and the TLS is associated with the thread, you could have two different coroutines access the same TLS in ways that violate borrow-checking rules; `std::thread::LocalKey::with` only provides immutable access as it is re-entrant. You only have a problem when you try to send the coroutine to another thread and then exit the original thread. If it stays on the same thread you just have multiple immutable references. If it goes to another thread but the old thread stays alive, you may simply access tls storage for the old thread for the current `LocalKey::with` call. ---
**Author:** ![RustyYato](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/rustyyato/32/13627_2.png) [@RustyYato](https://internals.rust-lang.org/u/RustyYato)\ **Post date:** [June 11, 2021, 10:19am UTC](https://internals.rust-lang.org/t/what-is-the-current-safety-story-for-library-based-stackful-coroutines/14863/11 "2021-06-11T10:19:33Z")
If you use interior mutability, like `Cell`, and swap to a different coroutine you could get two exclusive borrows at the same time. I.e. if you called `Cell::swap` on both coroutines. Because swapping coroutines could happen at any time (even during `Cell::swap`) ---
**Author:** ![bjorn3](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/bjorn3/32/2736_2.png) [@bjorn3](https://internals.rust-lang.org/u/bjorn3)\ **Post date:** [June 11, 2021, 2:45pm UTC](https://internals.rust-lang.org/t/what-is-the-current-safety-story-for-library-based-stackful-coroutines/14863/12 "2021-06-11T14:45:55Z")
> [@RustyYato](#): > > Because swapping coroutines could happen at any time (even during `Cell::swap` ) The library in question uses co-operative scheduling and not pre-emptive scheduling, so coroutine switching only happens at specific yield points. ---
**Author:** ![fweimer](https://avatars.discourse-cdn.com/v4/letter/f/d2c977/32.png) [@fweimer](https://internals.rust-lang.org/u/fweimer)\ **Post date:** [June 12, 2021, 5:59pm UTC](https://internals.rust-lang.org/t/what-is-the-current-safety-story-for-library-based-stackful-coroutines/14863/13 "2021-06-12T17:59:36Z")
Compilers cache the addresses of thread-local variables accessed within the same stack frame. After resumption on a different thread, the cache can be out of data and refer to TLS of the previous thread, even though the code is now running on a different thread. ---
**Author:** ![RalfJung](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/ralfjung/32/2415_2.png) [@RalfJung](https://internals.rust-lang.org/u/RalfJung)\ **Post date:** [June 13, 2021, 10:53am UTC](https://internals.rust-lang.org/t/what-is-the-current-safety-story-for-library-based-stackful-coroutines/14863/14 "2021-06-13T10:53:00Z")
> [@RustyYato](#): > > If you use interior mutability, like `Cell` , and swap to a different coroutine you could get two exclusive borrows at the same time. I.e. if you called `Cell::swap` on both coroutines. Because swapping coroutines could happen at any time (even during `Cell::swap` ) There are no `&mut` inside the implementation of `Cell::swap`, so even with preemptive scheduling this is not a problem I think. ---
**Author:** ![Soni](https://avatars.discourse-cdn.com/v4/letter/s/a3d4f5/32.png) [@Soni](https://internals.rust-lang.org/u/Soni)\ **Post date:** [June 13, 2021, 1:37pm UTC](https://internals.rust-lang.org/t/what-is-the-current-safety-story-for-library-based-stackful-coroutines/14863/15 "2021-06-13T13:37:48Z")
So why are these Send? ---
**Author:** ![atagunov](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/atagunov/32/5877_2.png) [@atagunov](https://internals.rust-lang.org/u/atagunov)\ **Post date:** [June 15, 2021, 12:09pm UTC](https://internals.rust-lang.org/t/what-is-the-current-safety-story-for-library-based-stackful-coroutines/14863/16 "2021-06-15T12:09:00Z")
> [@Soni](#): > > what's the issue of sharing TLS with things running on the same thread? I suspect assumptions made by unsafe and C code could get broken by cooperative multi-threading unless each co-routine has its own TLS and thus its own copy of thread-local variables Logical assumptions made by safe code may get broken too if more than a single co-routine has access to a thread-local variable. Suppose some library is using this mechanism to pass "hidden" parameters to functions for example.. > [@Soni](#): > > So why are these Send? I suppose it's same as for stackless co-routines. It's nice to be able to schedule a task on any OS-level thread from a pool rather than just on a single fixed OS-level thread. == So it seems there are two considerations here - when a stackful co-routine moves to a different executing OS-level thread (in a pool) it's nice to take thread-locals with it - when multiple stackful co-routines share the same executing OS-level thread it's nice if each has its own private copy of thread-local variables - there may be logical assumptions relying on this in both safe and unsafe code and in libraries written in C It seems rather appealing to be able to switch TLS at will and a library doing that might be a useful companion to `libpthread` both within and outside of Rust. ---
**Author:** ![Soni](https://avatars.discourse-cdn.com/v4/letter/s/a3d4f5/32.png) [@Soni](https://internals.rust-lang.org/u/Soni)\ **Post date:** [June 15, 2021, 12:27pm UTC](https://internals.rust-lang.org/t/what-is-the-current-safety-story-for-library-based-stackful-coroutines/14863/17 "2021-06-15T12:27:42Z")
> [@atagunov](#): > > It seems rather appealing to be able to switch TLS at will and a library doing that might be a useful companion to `libpthread` both within and outside of Rust. Sounds like you want something akin to [Lunatic](https://github.com/lunatic-solutions/lunatic). We'd rather have actually safe stackful coroutines, instead. But hey, to each their own. ^^ ---
**Author:** ![atagunov](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/atagunov/32/5877_2.png) [@atagunov](https://internals.rust-lang.org/u/atagunov)\ **Post date:** [June 18, 2021, 1:14pm UTC](https://internals.rust-lang.org/t/what-is-the-current-safety-story-for-library-based-stackful-coroutines/14863/18 "2021-06-18T13:14:47Z")
> [@josh](#): > > there's other code out there that manually allocates TLS @josh, would you know any from the top of you mind? examples of code doing that? ---
**Author:** ![Matthias247](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/matthias247/32/5734_2.png) [@Matthias247](https://internals.rust-lang.org/u/Matthias247)\ **Post date:** [July 2, 2021, 6:21am UTC](https://internals.rust-lang.org/t/what-is-the-current-safety-story-for-library-based-stackful-coroutines/14863/19 "2021-07-02T06:21:56Z")
Since it hasn't been brought up so far: The approach should be fine if the coroutines are not migrated between threads (1:N scheduling instead of M:N scheduling). Is that right? Might still be interesting for some applications then. ---
**Author:** ![atagunov](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/atagunov/32/5877_2.png) [@atagunov](https://internals.rust-lang.org/u/atagunov)\ **Post date:** [July 2, 2021, 4:19pm UTC](https://internals.rust-lang.org/t/what-is-the-current-safety-story-for-library-based-stackful-coroutines/14863/20 "2021-07-02T16:19:28Z")
> [@Matthias247](#): > > The approach should be fine if the coroutines are not migrated between threads ..? ...unless coroutines "contaminating" each other's thread locals is perceived as a problem? [Next page](https://internals.rust-lang.org/t/what-is-the-current-safety-story-for-library-based-stackful-coroutines/14863.md?page=2)