Restarting fixes Windows because your apps are broken

A server that sometimes plays games#

I started self hosting because of Immich, and I don’t have a server with a good GPU, so my laptop (a Legion 5 with a 3070) became the server. These days the same laptop also runs Jellyfin, sits on top of the TV unit with a gajillion wires going into the back, and never gets turned off.

After a couple days of being on, I realized I was getting really bad performance. This was back when I was using Cursor, and with Cursor I figured out that Nahimic (the audio “enhancer” Lenovo preinstalls) was leaking memory through audiodg.exe, and an nssm.exe was leaking too (I never found out which service registered that one). I removed both from the auto start lists, and later on disabled Nahimic completely:

Set-Service NahimicService -StartupType Disabled

If you want to check your own computer for leaks like these, sort your processes by memory and see if anything has gotten way bigger than makes sense, especially a process that has been running for days:

Get-Process | Sort-Object PM -Descending |
  Select-Object -First 5 Name, StartTime, @{ n = 'GB'; e = { [math]::Round($_.PM / 1GB, 1) } }

Air#

Most things were good after that. Then I wanted to use the laptop for gaming with friends and realized things were still not the best, so I restarted (of course), 3D printed a riser for airflow, since the Legion pulls air in from underneath, and cleaned the fins with an air gun. That did improve things, I just couldn’t tell you by how much, since I never thought of measuring any of this.

nvidia-smi will tell you why your GPU is slow#

I wish I had known this ten years ago: nvidia-smi can tell you whether your GPU is being throttled and why.

nvidia-smi --query-gpu=clocks_event_reasons.active,clocks.gr,temperature.gpu,power.draw --format=csv -l 2

clocks_event_reasons.active is a bitmask. 0x4 means the GPU hit its power limit (pretty normal on a laptop), and 0x20 means the GPU is too hot and is slowing itself down. I had no idea any of this existed, I just asked Claude Code why even a simple game like Peak was running badly, and Claude wrote a logger around that command and had me play for nine minutes:

300 sec  SW POWER CAP           pinned against the 115 W ceiling
228 sec  SW THERMAL SLOWDOWN    too hot to use even that
  0 sec  unthrottled

15:19:46  load starts        65 °C
15:25:53  thermal throttle   87 °C

Six minutes from cold to thermal throttling, the GPU clock dropping from 1,965 MHz to as low as 1,290, and not a single second where the GPU was running unthrottled. Claude’s first guess was dust or old thermal paste, but Peak was rendering at native 4K, because the laptop is plugged into a 4K TV and I never changed the resolution. A 115 W laptop GPU can’t do 4K, so I dropped the resolution to 1440p.

Two modes#

The chart is from later that night, running Subnautica 2 at 1440p. I had never played Subnautica like that, 60 FPS with maybe a few drops, even though the GPU was thermal throttling more than half the time. The CPU was the bad part, averaging 98.8 °C. The CPU and GPU share heatpipes in the Legion, so a hot CPU takes cooling away from the GPU, and since games on this laptop are limited by the GPU anyway, I capped the CPU at 35 W.

I had Claude create server mode and gaming mode scripts out of everything we learned. Server mode keeps the laptop quiet and cool and makes sure Jellyfin and friends are running. Gaming mode switches to the performance power plan, turns on the 15 W GPU boost, sets the fan curve and the CPU cap, and stops whatever is competing for the disk.

Two things I didn’t know before these scripts: powercfg doesn’t change the GPU power limit, the laptop’s embedded controller does, so the scripts set the mode there and check that the mode actually changed. And the power button was glowing red for “performance” while the GPU was still capped at 115 W out of 130 (nvidia-smi -q -d POWER shows the real limit).

Twenty days later, 2 FPS#

My friends and I found time to play again, so I pressed the gaming mode shortcut, opened Subnautica, and… 2 FPS?

Normally I would have just restarted, but my wife was watching a movie on Jellyfin and I couldn’t break her stream. So I told Opus 5.5 that the game was running at 2 FPS and that I couldn’t restart, and sent Opus on a quest to figure out why.

The movie#

Before I even opened the game the GPU was already at 100 W and 83 °C, the video decoder was 76% busy, the encoder 43%, and 2.8 GB of the 8 GB of VRAM was taken.

My wife had subtitles on, and the subtitles were PGS. The Jellyfin app on the iPad can’t render PGS (unlike text subtitles like SubRip), so Jellyfin was burning the subtitles into the video, which means decoding a 4K HDR movie, tone mapping, drawing the subtitles and re-encoding at 1080p. Subnautica 2 is an Unreal Engine 5 game that wants most of the 8 GB of VRAM, and my best guess for the 2 FPS is that Subnautica ran out of VRAM and spilled into system memory.

My wife was mostly listening and not really watching, so we turned the subtitles off and Jellyfin went back to direct playing the file. If you want to check whether something is transcoding on your GPU:

nvidia-smi --query-gpu=utilization.encoder,utilization.decoder,memory.used --format=csv -l 2

The game log says I was getting 21 to 33 FPS after that. We also noticed gaming mode had only half applied, because Windows had gone back to the Balanced power plan. With both fixed the FPS was okay, but the game was constantly stuttering and freezing, and last time I played, Subnautica was so smooth. None of this was making sense.

The overlay#

Windows had committed 48.6 GB of a 49.9 GB limit, and 12.7 GB of that was a single NVIDIA Overlay process that had been running for twenty days, since my last restart. (The same process was also reporting 58 TB of GPU memory on an 8 GB card, so NVIDIA’s accounting was clearly broken too.)

Stop-Process -Id 14080   # NVIDIA Overlay

NVIDIA starts a new overlay process right away, and the new one uses about 150 MB. Commit went down to 35.6 GB. I love that we found this one, a restart would have wiped the leak away and I would have never known the overlay was leaking. The game was still stuttering though.

Into the kernel#

Then I noticed the stuttering wasn’t just the game. Opus’s own terminal output was jumping and stuttering too, so the problem had to be something underneath everything. The System process was using between 1.1 and 1.4 cores nonstop.

Honestly, this is where I would have given up and restarted. I didn’t even know you could get a stack trace of the Windows kernel, but xperf from the Windows Performance Toolkit does exactly that (as admin):

xperf -on PROC_THREAD+LOADER+PROFILE -stackwalk Profile
Start-Sleep 10
xperf -d trace.etl

$env:_NT_SYMBOL_PATH = 'srv*C:symbols*https://msdl.microsoft.com/download/symbols'
xperf -i trace.etl -symbols -a stack -pid 4 -butterfly 50   # pid 4 is System

1. dxgmms2.sys!VidMmWorkerThreadProc 75.81%

   The kernel thread that manages GPU memory for every program.

2. dxgmms2.sys!VIDMM_GLOBAL::HandleTrimWnf 75.28%

   GPU memory is tight (a game using 7 of 8 GB will do that), so processes get asked to shrink.

3. ntkrnlmp.exe!NtUpdateWnfStateData 74.68%

   The request goes out through WNF, the kernel's publish/subscribe system.

4. ntkrnlmp.exe!ExpWnfFindScopeInstance 74.30%

   Publishing needs the right WNF scope, and the scope is found by walking a list,

5. ntkrnlmp.exe!memcmp 15.97%

   comparing entries one by one.

Every window, the game and the terminal included, gets drawn into GPU memory, and the Desktop Window Manager combines those windows into the frame on your screen. A single kernel thread manages GPU memory for all programs, and that thread was spending three quarters of its time walking a list. While that thread is walking the list, everything that needs GPU memory has to wait, so the whole screen was hitching and not just the game.

32,709 zombie processes#

A zombie is a process that has exited but can’t be cleaned up by Windows, because some other program still holds a handle to the process. Opus compared the number of process objects in the kernel against the number of processes that were running:

Live processes: 306   Process objects in kernel: 33,015   => zombies ~ 32,709

Name                    HandleCount
GameManagerService            33695
svchost                       29855
Razer Synapse Service         28193

Razer Game Manager watches every process you launch to detect games, and seems to open a handle for every process without ever closing them. Twenty days without a restart comes out to about 1,600 new zombies a day. The software that makes my mouse glow was slowing down every frame on my screen, great.

The bar under kernel on the laptop’s screen is that list drawn to scale, and the game freezes whenever the list gets walked.

Stop-Service 'Razer Game Manager Service', 'Razer Synapse Service' -Force

Within a minute the zombies went from 32,709 to 260, the System process went from about 1.25 cores to 0.2, and the stuttering was gone. No restart, and my wife didn’t even notice.

(Get-Counter 'ObjectsProcesses').CounterSamples[0].CookedValue - (Get-Process).Count

Get-Process | Sort-Object HandleCount -Descending | Select-Object -First 5 Name, HandleCount