# Solved: \`Thread::unpark\` is extremely slow on windows

**URL:** <https://internals.rust-lang.org/t/solved-thread-unpark-is-extremely-slow-on-windows/11698>\
**Category:** Uncategorized\
**Created:** [January 25, 2020, 2:34pm UTC](https://internals.rust-lang.org/t/solved-thread-unpark-is-extremely-slow-on-windows/11698 "2020-01-25T14:34:06Z")\
**Posts on this page:** 16\
**Page:** 1

<div class="post-metadata">

**Author:** ![matklad](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/matklad/32/12266_2.png) [@matklad](https://internals.rust-lang.org/u/matklad)\
**Post date:** [January 25, 2020, 2:34pm UTC](https://internals.rust-lang.org/t/solved-thread-unpark-is-extremely-slow-on-windows/11698/1 "2020-01-25T14:34:06Z")

</div>

I am investigating a particular instance of slowness in rust-analyzer, and, debugging through a threadpool and crossbeam-channel, I made the following observation:

- If I pin my program to one core with

```rust
unsafe {
    let process = winapi::um::processthreadsapi::GetCurrentProcess();
    let mask = 1;
    winapi::um::winbase::SetProcessAffinityMask(process, mask);
}

```

- then I repeatedly observe `std::thread::Thread::unpark` taking 30-40 **milliseconds**.

Specifically, sticking `Instant::now/::elapsed` around [this](https://github.com/crossbeam-rs/crossbeam/blob/ebecb82c740a1b3d9d10f235387848f7e3fa9c68/crossbeam-channel/src/context.rs#L182-L184) routinely leads to printouts with `unpark = 39.9091ms` or some such ([example](https://gist.github.com/matklad/92085b5eb7e317cd55fd8f336f2a0b89)). This seems very surprising to me, I would expect `unpark` to be sys-call lenght (hundreds of nanos), but this waaaay longer.

Now, this is the bottom of the rabbit hole (cc @retep998 ) for me for now, as I can't `[patch.crates-io]` libstd easily. Does anyone perchance know what might be happening here?

Note that I don't have a minimal reproducible example yet, as it requires patching rust-analyzer, threadpool and crossbeam.

EDIT: @jstarks figured it out, it were priority boosts: [Solved: `Thread::unpark` is extremely slow on windows](https://internals.rust-lang.org/t/thread-unpark-is-extremely-slow-on-windows/11698/13)

---

<div class="post-metadata">

**Author:** ![Diggsey](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/diggsey/32/6311_2.png) [@Diggsey](https://internals.rust-lang.org/u/Diggsey)\
**Post date:** [January 25, 2020, 2:46pm UTC](https://internals.rust-lang.org/t/solved-thread-unpark-is-extremely-slow-on-windows/11698/2 "2020-01-25T14:46:34Z")

</div>

I don't know why this is happening, but this whole "optimization" seems suspicious:

> <https://github.com/rust-lang/rust/blob/master/src/libstd/thread/mod.rs#L1191-L1209>

I suspect the overhead of this extra lock would outweigh the cost of just always calling `notify_one`, which on windows translates to a call to `WakeConditionVariable`, which AFAIK doesn't always result in a syscall. Still it doesn't really explain why it's _so_ slow.

---

<div class="post-metadata">

**Author:** ![matklad](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/matklad/32/12266_2.png) [@matklad](https://internals.rust-lang.org/u/matklad)\
**Post date:** [January 25, 2020, 3:07pm UTC](https://internals.rust-lang.org/t/solved-thread-unpark-is-extremely-slow-on-windows/11698/3 "2020-01-25T15:07:07Z")

</div>

Is there any workflow to use a patched version of stdlib? I’d love to add more printfs...

---

<div class="post-metadata">

**Author:** ![Amanieu](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/amanieu/32/4095_2.png) [@Amanieu](https://internals.rust-lang.org/u/Amanieu)\
**Post date:** [January 25, 2020, 3:18pm UTC](https://internals.rust-lang.org/t/solved-thread-unpark-is-extremely-slow-on-windows/11698/4 "2020-01-25T15:18:31Z")

</div>

It's a bit of a hack, but you can modify your rust-src in ~/.rustup and then use xargo to build a custom libstd.

---

<div class="post-metadata">

**Author:** ![matklad](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/matklad/32/12266_2.png) [@matklad](https://internals.rust-lang.org/u/matklad)\
**Post date:** [January 25, 2020, 3:53pm UTC](https://internals.rust-lang.org/t/solved-thread-unpark-is-extremely-slow-on-windows/11698/5 "2020-01-25T15:53:10Z")

</div>

I am stumped:

```rust
    #[inline]
    pub unsafe fn notify_one(&self) {
        let s = crate::time::Instant::now();
        c::WakeConditionVariable(self.inner.get());
        eprintln!("sys/notify_one {:?}", s.elapsed());
    }

```

The above prints `sys/notify_one 35.3309ms`. `WakeConditionVariable` is "libc" function (there was a bit of logic to load fallback for win XP, but I've removed it in favor of just unconditionally calling the function) .

---

<div class="post-metadata">

**Author:** ![jstarks](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/jstarks/32/6296_2.png) [@jstarks](https://internals.rust-lang.org/u/jstarks)\
**Post date:** [January 25, 2020, 4:25pm UTC](https://internals.rust-lang.org/t/solved-thread-unpark-is-extremely-slow-on-windows/11698/6 "2020-01-25T16:25:10Z")

</div>

Which version of Windows are you testing against?

---

<div class="post-metadata">

**Author:** ![matklad](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/matklad/32/12266_2.png) [@matklad](https://internals.rust-lang.org/u/matklad)\
**Post date:** [January 25, 2020, 4:25pm UTC](https://internals.rust-lang.org/t/solved-thread-unpark-is-extremely-slow-on-windows/11698/7 "2020-01-25T16:25:55Z")

</div>

Windows 10 Pro

---

<div class="post-metadata">

**Author:** ![Diggsey](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/diggsey/32/6311_2.png) [@Diggsey](https://internals.rust-lang.org/u/Diggsey)\
**Post date:** [January 25, 2020, 4:45pm UTC](https://internals.rust-lang.org/t/solved-thread-unpark-is-extremely-slow-on-windows/11698/8 "2020-01-25T16:45:51Z")

</div>

Oh, I think I know why this is happening. You are limiting the process to a single logical processor, and when you call `thread::unpark`, it is context switching away from the current thread (giving up its remaining time-slice). If other threads in the process are CPU-bound, they will use up their entire time-slices (15-20ms each by default) so what you are seeing corresponds to about two CPU-bound threads in the same process.

---

<div class="post-metadata">

**Author:** ![matklad](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/matklad/32/12266_2.png) [@matklad](https://internals.rust-lang.org/u/matklad)\
**Post date:** [January 25, 2020, 5:01pm UTC](https://internals.rust-lang.org/t/solved-thread-unpark-is-extremely-slow-on-windows/11698/9 "2020-01-25T17:01:52Z")

</div>

But _why_ would it context switch away? This is `unpark`, not `park`. I definitely don't want to context switch away. The thing I am doing, on the high-level, is `threadpool.execute(move || some_task())`. That `execute` uses a channel to send the task to the worker thread, and that channel uses `unpark` to actually wake the worker. I want the main thread to continue doing the work (which it definitely can do).

---

<div class="post-metadata">

**Author:** ![Diggsey](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/diggsey/32/6311_2.png) [@Diggsey](https://internals.rust-lang.org/u/Diggsey)\
**Post date:** [January 25, 2020, 5:19pm UTC](https://internals.rust-lang.org/t/solved-thread-unpark-is-extremely-slow-on-windows/11698/10 "2020-01-25T17:19:50Z")

</div>

If the two threads are the same priority why would it _not_ switch to the thread that's being woken up? The thread being unparked also has work it can do.

---

<div class="post-metadata">

**Author:** ![matklad](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/matklad/32/12266_2.png) [@matklad](https://internals.rust-lang.org/u/matklad)\
**Post date:** [January 25, 2020, 5:29pm UTC](https://internals.rust-lang.org/t/solved-thread-unpark-is-extremely-slow-on-windows/11698/11 "2020-01-25T17:29:22Z")

</div>

The good reason not to switch is fairness. In my case, the main thread spawns ten background tasks in a row, and then actually cancels them. If `unpark` also yields, my main thread doesn't use a full quant, while all background threads do. But I see how OS API _could_ be implemented that way.

---

<div class="post-metadata">

**Author:** ![Diggsey](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/diggsey/32/6311_2.png) [@Diggsey](https://internals.rust-lang.org/u/Diggsey)\
**Post date:** [January 25, 2020, 5:38pm UTC](https://internals.rust-lang.org/t/solved-thread-unpark-is-extremely-slow-on-windows/11698/12 "2020-01-25T17:38:45Z")

</div>

I don't believe the windows scheduler is completely fair, but you should be able to lower the priority of the thread pool to prevent it from conflicting (assuming the CPU bounds tasks are running on the thread pool).

---

<div class="post-metadata">

**Author:** ![jstarks](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/jstarks/32/6296_2.png) [@jstarks](https://internals.rust-lang.org/u/jstarks)\
**Post date:** [January 25, 2020, 7:50pm UTC](https://internals.rust-lang.org/t/solved-thread-unpark-is-extremely-slow-on-windows/11698/13 "2020-01-25T19:50:13Z")

</div>

Internally in Windows, WakeConditionVariable dynamically (temporarily) boosts the waiting thread's priority by one, which likely makes it higher priority than the waker thread. Therefore it runs first.

There's a little info on priority boosts at [https://docs.microsoft.com/en-us/windows/win32/procthread/priority-boosts](https://docs.microsoft.com/en-us/windows/win32/procthread/priority-boosts).

---

<div class="post-metadata">

**Author:** ![Amanieu](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/amanieu/32/4095_2.png) [@Amanieu](https://internals.rust-lang.org/u/Amanieu)\
**Post date:** [January 26, 2020, 12:36am UTC](https://internals.rust-lang.org/t/solved-thread-unpark-is-extremely-slow-on-windows/11698/14 "2020-01-26T00:36:35Z")

</div>

You could try disabling the priority boost to see if this help: `SetProcessPriorityBoost(GetCurrentProcess(), FALSE)`

---

<div class="post-metadata">

**Author:** ![matklad](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/matklad/32/12266_2.png) [@matklad](https://internals.rust-lang.org/u/matklad)\
**Post date:** [January 26, 2020, 10:48am UTC](https://internals.rust-lang.org/t/solved-thread-unpark-is-extremely-slow-on-windows/11698/15 "2020-01-26T10:48:29Z")

</div>

> [@Amanieu](#):
>
> SetProcessPriorityBoost(GetCurrentProcess(), FALSE)

It should be `TRUE`, because

```C++
BOOL SetProcessPriorityBoost(
  HANDLE hProcess,
  BOOL bDisablePriorityBoost
);

```

But yes, I confirm that this does in fact fix the problem I am seeing, priority boosts are to blame!

---

<div class="post-metadata">

**Author:** ![system](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/system/32/14092_2.png) [@system](https://internals.rust-lang.org/u/system)\
**Post date:** [April 25, 2020, 10:48am UTC](https://internals.rust-lang.org/t/solved-thread-unpark-is-extremely-slow-on-windows/11698/16 "2020-04-25T10:48:37Z")

</div>

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.