Proxmox LXC with QuickSync 100% cpu/crashes

[slykens]

Hello All -

Looking for some ideas here. I have Channels running in a container on proxmox mostly happily. In my case it is a privileged container because my storage is on NFS. I will eventually rebuild the container and just expose the storage from the host but this is what I've got right now. I don't think privileged vs. unprivileged is an issue here but mention it for completeness.

Sometimes when Channels starts a transcode for a remote user using QuickSync, CPU will spike to 100% and channels essentially crashes. I have also had the container just plain shut down, as well. It does not crash every time a transcode is requested. After the restart here I had a transcode run for about two hours straight without issue.

Channels does not log anything interesting when this happens - I see a log entry to start encoding then the startup log, for example:

2024/10/12 15:39:49.292603 [TNR] Opened connection to M3U-412OTA for ch2.1 CBS
2024/10/12 15:39:49.292929 [HLS] Starting live stream for channel 2.1 from (remote IP)
2024/10/12 15:39:51.189403 [HLS] Probed live stream in 1.873537352s: mpeg2video 1920x1080 tt 7508008bps
2024/10/12 15:39:52.960713 [HLS] Session ch2.1-dANY-89fa536ba879 started in 3.667750784s
2024/10/12 15:39:53.083705 [ENC] Starting encoder for ch2.1 in /data/Streaming/ch2.1-dANY-89fa536ba879-3653241923/encoder-1-2120336797 at 1 (2.256033) (encoder=h264_vaapi, codec=h264, acodec=aac, resolution=1080, deinterlacer=hardware, bitrate=7488, segment_size=0.01)
2024/10/12 15:53:25.393135 [SYS] Starting Channels DVR v2024.09.10.2115 (linux-x86_64 pid:157) in /usr/local/channels-dvr/data
2024/10/12 15:53:25.415666 [SYS] Started HTTP Server on 8089
2024/10/12 15:53:38.856780 [HDR] Found 1 devices

If it matters, in this particular example the source is another Channels server in a datacenter that exposes its OTA channels by M3U to my Channels at home. I have seen the same behavior, however, using local OTA sources as well.

Looking for some ideas or guidance - anywhere I can increase logging verbosity and maybe pick up a pointer to the problem.

Thanks.

[Dxcv]

Might be worth testing out the source from the other server in HLS instead of Mpeg TS. HLS tends to be more forgiving for packet latency etc.

[bnhf]

@slykens

Sometimes when Channels starts a transcode for a remote user using QuickSync, CPU will spike to 100% and channels essentially crashes

What resources do you have allocated to the LXC?

What other VMs and LXCs are running at the same time?

Any chance you've over allocated RAM across your virtualizations?

[slykens]

I don't see any packet loss generally for this connection but since it also fails with a local HDHR tuner I don't think that's the issue.

[slykens]

I have set to 512 MB with 2 GB swap. I don't see it exhaust memory but could kick it up to 2-4 GB to be sure. The host has lots of available resources - only using 51 GB of 128 GB RAM and usually only about 2-3% cpu on the host at any time.

No other container is sharing the GPU. There's eight containers and six VMs on this host, none resource intensive.

[bnhf]

@slykens

2024/10/12 15:39:53.083705 [ENC] Starting encoder for ch2.1 in /data/Streaming/ch2.1-dANY-89fa536ba879-3653241923/encoder-1-2120336797 at 1 (2.256033) (encoder=h264_vaapi, codec=h264, acodec=aac, resolution=1080, deinterlacer=hardware, bitrate=7488, segment_size=0.01)

What kind of disk space do you have available in the partition where /data is located?

[slykens]

@bnhf

What kind of disk space do you have available in the partition where /data is located?

3.5 TB.

[bnhf]

@slykens I've been testing a new ultra-compact NAS the last couple of days with Proxmox, with the idea of using it in conjunction with my existing Proxmox-based rackmount setup.

Anyway, your issue came to mind as something I could use both to test the new NAS, and attempt to duplicate your setup.

Nothing much to report so far, as I've been transcoding for about 4 hours now, and everything has been working as expected. This test host setup has a 12th gen i3 with 16GB of RAM and a 4TB NVMe drive.

The CDVR LXC I'm running has 2 vCPUs allocated, 2GB of RAM and 1TB of disk. It's running Proxmox 8.2.7, with /dev/dri/card0, /dev/dri/renderD128 and /dev/net/tun all passed through using the new option for that purpose in the LXC's Resources.

Here are a few screenshots of what I'm seeing with transcoding active:

This is an unprivileged LXC, as I never run privileged, with the virtual channel I'm using for testing delivering 1080p content from a Samba share.