Specific problem with having the OS and the game itself on different drives (FSR)

Windows on C: and Poe 2 on D:

When you open the game and play it (with FSR on) it's fine at first, but when you close the game and re-open it again, the gpu usage is at high % of usage even on the login screen, and in-game the gpu is at 99% usage even on your own hideout

My theory is that the game is either rendering 2 times, or some bug caused by shader cache location being in different location than the OS

My system:

Windows 10 22h2
Ryzen 5 3600
RX 6600 8GB
16GB RAM

Game settings:
API: DX12
Texture Quality: High
Texture Filtering: x8
Lightning: Shadows + Global Ilumination
Shadow + GI: Low
Sun Shadow Quality: Low
Number of Lights: Medium

AMD Driver version: 25.9.1