Environment:
- OS: Debian/Ubuntu LXC in Proxmox 9.1
- GPU: NVIDIA RTX 5060 Ti (Blackwell)
- NVIDIA Driver: 595.58.03
- CDVR FFmpeg: 6.0.2
Issue:
Standard NVENC hardware transcoding works normally. However, any pipeline requiring hardware tone mapping (e.g., hwupload_cuda -> scale_cuda -> h264_nvenc) fails immediately with NV_ENC_ERR_INVALID_PARAM.
Possible Cause:
The bundled static FFmpeg 6.0.2 appears to be compiled against older nv-codec-headers. It seems the headers fail to properly register the input hardware frame/memory layout required by the newer NVENC v13.0 API on Blackwell GPUs.
Testing:
When executing the same pipeline in the LXC using a current ffmpeg (master-latest-linux64-gpl), the encode succeeds at >160fps.
Failing h264_nvenc+tonemap:
h264_nvenc+tonemap
[Parsed_testsrc_0 @ 0x31cd63c0] size:1280x720 rate:60/1 duration:-1.000000 sar:1/1
[auto_scale_0 @ 0x31cd8d40] w:iw h:ih flags:'' interl:0
[Parsed_format_1 @ 0x31cd6780] auto-inserting filter 'auto_scale_0' between the filter 'Parsed_testsrc_0' and the filter 'Parsed_format_1'
[auto_scale_1 @ 0x31ce6a00] w:iw h:ih flags:'' interl:0
[Parsed_colorspace_2 @ 0x31cd6a00] auto-inserting filter 'auto_scale_1' between the filter 'Parsed_format_1' and the filter 'Parsed_colorspace_2'
[auto_scale_0 @ 0x31cd8d40] w:1280 h:720 fmt:rgb24 sar:1/1 -> w:1280 h:720 fmt:p010le sar:1/1 flags:0x00000004
[auto_scale_1 @ 0x31ce6a00] w:1280 h:720 fmt:p010le sar:1/1 -> w:1280 h:720 fmt:yuv420p10le sar:1/1 flags:0x00000004
Last message repeated 3 times
Input #0, lavfi, from 'testsrc=size=1280x720:rate=60,format=p010le,colorspace=all=bt2020:iall=bt2020:fast=0':
Duration: N/A, start: 0.000000, bitrate: N/A
Stream #0:0: Video: wrapped_avframe, 1 reference frame, yuv420p10le(bt2020nc/bt2020/smpte2084), 1280x720 [SAR 1:1 DAR 16:9], 60 fps, 60 tbr, 60 tbn
Stream mapping:
Stream #0:0 -> #0:0 (wrapped_avframe (native) -> h264 (h264_nvenc))
Press [q] to stop, [?] for help
[graph 0 input from stream 0:0 @ 0x335ab8c0] w:1280 h:720 pixfmt:yuv420p10le tb:1/60 fr:60/1 sar:1/1
[auto_scale_0 @ 0x335ad240] w:iw h:ih flags:'' interl:0
[trim_in_0_0 @ 0x335abcc0] auto-inserting filter 'auto_scale_0' between the filter 'graph 0 input from stream 0:0' and the filter 'trim_in_0_0'
[auto_scale_0 @ 0x335ad240] w:1280 h:720 fmt:yuv420p10le sar:1/1 -> w:1280 h:720 fmt:p010le sar:1/1 flags:0x00000004
Last message repeated 3 times
[h264_nvenc @ 0x32685e40] Using input frames context (format cuda) with h264_nvenc encoder.
[h264_nvenc @ 0x32685e40] Loaded Nvenc version 13.0
[h264_nvenc @ 0x32685e40] Nvenc initialized successfully
Output #0, null, to '/dev/null':
Metadata:
encoder : Lavf60.3.100
Stream #0:0: Video: h264 (High), 1 reference frame, cuda(tv, bt709, progressive), 1280x720 (0x0) [SAR 1:1 DAR 16:9], q=2-31, 4000 kb/s, 60 fps, 60 tbn
Metadata:
encoder : Lavc60.3.100 h264_nvenc
Side data:
cpb: bitrate max/min/avg: 6000000/0/4000000 buffer size: 8000000 vbv_delay: N/A
[h264_nvenc @ 0x32685e40] Error registering an input resource: invalid param (8):
[h264_nvenc @ 0x32685e40] Could not register an input HW frame
[vost#0:0/h264_nvenc @ 0x32685b40] Error submitting video frame to the encoder
[out#0/null @ 0x32685440] All streams finished
[out#0/null @ 0x32685440] Terminating muxer thread
[h264_nvenc @ 0x32685e40] Nvenc unloaded
Terminating demuxer thread 0
Conversion failed!
Successful Manual Test:
root@channelsdvr:/tmp/ffmpeg-master-latest-linux64-gpl/bin# ./ffmpeg -hide_banner -init_hw_device cuda=cu:0 -filter_hw_device cu -v info -f lavfi -i testsrc -t 1 -f null -
Input #0, lavfi, from 'testsrc':
Duration: N/A, start: 0.000000, bitrate: N/A
Stream #0:0: Video: wrapped_avframe, rgb24, 320x240 [SAR 1:1 DAR 4:3], 25 fps, 25 tbr, 25 tbn
Stream mapping:
Stream #0:0 -> #0:0 (wrapped_avframe (native) -> wrapped_avframe (native))
Press [q] to stop, [?] for help
Output #0, null, to 'pipe:':
Metadata:
encoder : Lavf62.13.102
Stream #0:0: Video: wrapped_avframe, rgb24(pc, gbr/unknown/unknown, progressive), 320x240 [SAR 1:1 DAR 4:3], q=2-31, 200 kb/s, 25 fps, 25 tbn
Metadata:
encoder : Lavc62.29.101 wrapped_avframe
[out#0/null @ 0x5ae355704a80] video:10KiB audio:0KiB subtitle:0KiB other streams:0KiB global headers:0KiB muxing overhead: unknown
frame= 25 fps=0.0 q=-0.0 Lsize=N/A time=00:00:01.00 bitrate=N/A speed=47.2x elapsed=0:00:00.02
root@channelsdvr:/tmp/ffmpeg-master-latest-linux64-gpl/bin# ./ffmpeg -v error -stats \
-f lavfi -i "testsrc=size=1280x720:rate=60,format=p010le" \
-vf "hwupload_cuda,scale_cuda=format=yuv420p" \
-c:v h264_nvenc \
-f null -
frame=14035 fps=167 q=11.0 Lsize=N/A time=00:03:53.86 bitrate=N/A speed=2.78x elapsed=0:01:24.03
root@channelsdvr:/tmp/ffmpeg-master-latest-linux64-gpl/bin#
Does an update need to be made on the CDVR ffmpeg or is there anything else I can modify/test to see if I can fix it locally?