What I discovered is Widevine decryption is CPU bound so passing through the GPU thru works but any other activity on the server creates latency. Confirmed on two Unraid boxes with my old Quadro P400. Eventually video will go slow on and audio will sound like Darth Vader.
What did help was isolating a few cores and pinning them but it still wasn't a great experience for stream stability. I moved to an old Kaby Lake mini itx build my grandparents recently retired (that I built for them nearly 10 years ago) and CH4C is running super well with the P400 in that on bare metal.
Virtualized Alder Lake is worse than bare metal Kaby Lake for this workload. I also noticed high dpc latency in this VM which likely contributed. The nice thing about this machine is 5 display outputs in one SFF box and Windows 11 IoT Enterprise LTSC runs great on it with a decade of patches. I do wonder if anything can be done to make CH4C less sensitive to latency or if it's Chrome.
Edit: I was passing the card through, the issue was widevine software decoding.