Why isn't my process terminating on Windows on ARM?

This is the first post in a series from MIND's R&D team in Tel Aviv, where we write up the harder problems we run into building the agent behind Stress-Free DLP. This one is about a process that wouldn't die, and the Windows on ARM emulation bug underneath it.

At MIND we develop an agent for both Windows and macOS. To test the Windows agent, we use VMs that run Windows on ARM. During development, we noticed that our process sometimes hangs while terminating.

After it happened a few times, I started wondering why it only happens to our processes. The first thing I did was search for reports of similar issues. I found a few that describe the same symptoms, but none of them explained why it happens. So I did what I like most. I took a live kernel dump.

Inspecting the hung process:

The stuck thread is hanging inside NtGetContextThread. Let's look at the thread whose context it's trying to get. It's stored in x19 on the NtGetContextThread frame.

The target thread is TERMINATED. So why does NtGetContextThread hang? GetThreadContext is supposed to return an error when it's called on a terminated thread. Let's dive into what NtGetContextThread actually does. Most of the magic happens in PspGetContextThreadInternal. Here is the block where the interesting part happens.

Disclaimer: I used AI to name most of the variables here, so some names may be inaccurate.

I highlighted the parts that run in the normal flow. Here is the cleaned-up code of GetContext in that flow:

To understand how this code works, you first have to know what an APC is.

An APC is an object that lets you run a function in the context of a specified thread. For example, the kernel uses APCs to complete IO calls and copy the requested data into the process memory. Terminating a thread is also done through an APC. There are several types of APCs, which you can read about in the Microsoft docs. They differ by the context the APC runs in (user mode or kernel mode) and by which code they can preempt.

PspGetContextThreadInternal uses a special kernel APC (as you can see in the name PspGetSetContextSpecialApc), which means it can preempt any code running at PASSIVE_LEVEL, including other APCs. Once you know what an APC is, you can see why it's perfect for GetThreadContext. Just queue an APC on the target thread, and then, while running on that thread, read the context you just preempted (more precisely, you have to read the user-mode context that was preempted). When PspGetSetContextSpecialApc finishes, it signals the event that was passed to it in the context.

So far it looks good, but when our code runs it actually enters the following if block.

It enters this if block only if the fourth bit is set in OpFlags. If you look a little above, you can see exactly when that bit is set.

Looking at ((byte)(CurrentThread->Header).field0_0x0._s_6.field3_0x3._s_1 >> 6 & 1) != 0 (notice that it checks this bit on the target thread earlier), it checks the 7th bit of one of the fields in the DISPATCHER_HEADER. Here is that field for our thread in WinDbg:

You can see that the 7th bit actually tells whether the thread is emulated. That makes sense, because we sometimes test our production agent, which we compile for x86-64.

So in our process, the flow that runs looks like this:

So if the current thread is emulated, it first waits up to 10 seconds with a UserMode wait. If that wait doesn't succeed, it queues a cancel APC to cancel the current operation, clears current_thread_emulated from the operation flags and tries to get the context again. Bottom line:

• 1st try: current_thread_emulated=true, wait up to 10 seconds with a UserMode wait
• 2nd try: current_thread_emulated=false, wait indefinitely with a KernelMode wait

If you look carefully, on the second attempt it waits twice. Once on line 29 if the APC was inserted, and again on line 33 even if the insertion failed.

So here is what happened to our agent's process. Thread A calls GetThreadContext on thread B and reaches line 12. Meanwhile, TerminateProcess is called on the process, and thread B is terminated.

The wait on line 12 is a UserMode wait, which means it can return STATUS_USER_APC when someone terminates the thread, and that's exactly what happens here. The wait on line 12 fails, so we go into the 2nd try. But thread B is already terminated, so no new APCs can be queued for it. The APC is never queued, yet we still wait for finish_event on line 33. That wait is a KernelMode wait, so it won't return even though the current thread is terminating.

With Claude's help, I quickly wrote code that reproduces this exact case:

The repro was running on Windows 25H2 26200.8655.

Our agent is written in Go. One of Go's core features is goroutines, lightweight user-mode threads. Goroutine scheduling is mostly cooperative, meaning a goroutine has to voluntarily give up its execution time to let other goroutines run on the current thread. To avoid goroutine starvation, the Go runtime spawns a sysmon thread that preempts goroutines running for too long without yielding their thread (for example, a heavy computation loop). It does that by injecting a call to the preemption function into the thread (Go runtime source). So basically it does GetThreadContext(), edits the context's PC, SP, etc., then SetThreadContext() on its own threads.

This happens very frequently when our process runs emulated as x64 on an ARM VM, so terminating it has a high chance of hitting the race.

I didn't dig deep into this, but here's my guess. Because Windows emulates the x64 instructions, the special APC can be queued in the middle of an emulated instruction. The first APC tries to get the context only at an instruction boundary. If that doesn't work, it falls back to an APC that could land while an instruction is emulated.

The fix is simple. Don't call KeWaitForSingleObject if the APC insertion failed. I patched the code in a debugger so it jumps over the second KeWaitForSingleObject when the insertion fails, and it worked. There isn't a real workaround until Microsoft fixes it.

This is the kind of bug we chase so the MIND agent stays quiet and reliable on every platform our customers run. Stress-Free DLP for them means work like this for us: minding the low-level details so the product above them just works. If kernel dumps and emulation edge cases sound like a good week, our R&D team in Tel Aviv is hiring. See our open roles.

• Types of APCs, Microsoft Learn
• Go runtime, preemptM in os_windows.go