OMX Hardware Transcoding on Raspberry PI

Channels DVR Server
#53

v4l2 has a scaling API, but the broadcom driver does not implement it. There's no deinterlacing available in the hardware.

I pulled some patches earlier which should fix this. I'm uploading a new build now with those included. Let me know if you still see stalling after that.

Awesome. Both h264 and mpeg2 720p sources do very well in my tests.

The ISP component is complicated to use, but the RESIZE one seems like it would give us hardware scaling and remove the major bottleneck. I'll take a stab at it although I don't know if I have the time/patience required to actually pull it off.

#54

I wrote a basic hardware scaler. Performance is improved, but still not reaching 30fps to do real-time encode.

720p: swscaler=12fps fastswscaler=19fps hwscaler=24fps
480p: swscaler=13fps fastswscaler=27fps hwscaler=29fps

My initial approach is very naive and copies the video frame out of the GPU rescaler and then back into the GPU encoder, so there are still some more performance gains possible.

#55

Finally got my Pi4 today!

The latest build runs very nicely on it. Getting 1.2x converting 1080i mpeg2 -> 720p h264.

2 Likes
#56

Very nice. That's with your new hardware scaler? Sounds like you must have further improved its performance. I'll give it a try. BTW, I'm not able to view remotely in Channels app with the most recent betas, only in the browser. Remotely the app just says "Your Channels DVR computer is not powerful enough".

BTW, did your scaler go the v4l2 route? I'm sure lots of people would appreciate playing with that (thinking PI-based security cameras, home automation video monitoring systems, etc., where you want to scale for mobile live feeds).

#57

Yea this is with the scaler. It uses the ISP hardware via V4L2. I had been testing on the RPI3+ where the perf is still not great, but the RPI4 is so much faster it doesn't matter.

Will fix.

#58

Hey this is fantastic! Tried 2019.08.27.0251 remotely in the web interface, and it works really well for 1080i content. With my weak 5mpbs upload speed, I had to dial it back to 720p3mbps. Monitoring channels-dvr upload usage revealed about 350-420KB/s or 2.8-3.3Mbps, so it seems the hardware encoder is respecting the streaming rate. Quality is decent. It loads pretty quickly (5-10s) and is quite stable at the 720p3mbps (or 4mpbs) setting for 1080i content. Thanks to your hardware scaler, CPU stays below 150%, and often less than 100%. That means the Pi will have plenty of time to do other things. Truly remarkable progress here!

Unfortunately 720p content is a different story — it either stalls regularly, or refuses to start playing entirely. While attempting to stream 720p content, CPU is humming along at 125% or less (seems about right), but what's strange is it's completely saturating my upstream pipe at ~570KB/s for minutes at a time. Kind of seems like it isn't respecting the data rate limits for 720p content when the scaler isn't used.

Interestingly 480p also seems close to correct for upload speed (assuming a 720p 4mbps corresponds to about 1.8mpbs 480p). I would guess you don't "upscale" so not sure why 720p has such specific issues; maybe some asymmetry in how the encoder is setup with and without scaling. Also odd because 720p content used to be the easiest!

Anyway, great progress. Looking forward to seeing how this evolves.

2 Likes
#59

Can you try 720p again with the latest build? I'm not seeing the issues you were having, but maybe it got fixed along the way.

You should also be able to stream from the Channels app now.

EDIT: Never mind, I see that bitrate is not being respected for some reason.

#60

@jdts Your analysis was correct in that bitrate controls were not working in some cases. The issue was not related to the resolution, but rather the framerate of the video source.

The issue is resolved in the latest build, so you should be able to transcode and stream all channels and recordings, including to the Channels app.

#61

Great, 720p4mpbs runs about 4-5mbps for either 30 or 60fps content now. And CPU is always below 150%, sometimes below 100%. Clearly no overclocking is needed. This is really fantastic progress in such a short time. Did you end up figuring out how to keep the scaled frame in the GPU for encoding?

I think with a bit more testing (including h264 content, which I don't have in my setup), the Pi4 would be an easy recommend for a highly affordable Channels DVR server.

1 Like
#62

Playing with TVE on my Pi4. I was surprised to see ffmpeg occasionally pushing up to 200% in the web client on lower bitrates (<=4mbps), dropping to ~0% if you up the bitrate to 6mbps (720p in and out). Presumably that's because it has to decode and reencode at lower than native bitrate.

Couple questions:

  1. Does a channels client (e.g. on iOS) used remotely actually take the trouble to downlink/uplink a given video through the DVR at home? If so, it would seem more sensible to just go directly to the TVE source and skip the middleman.
  2. Is TVE h.264 decoding being hardware accelerated? I think omx supports that.
#63

Yes it does. The reason is that the source stream is authenticated by the DVR server, and so that is the original "consumer" of the TVE stream. Also, TVE streams are always retrieved at full resolution, so downscaling only occurs at the DVR.

I believe I remember hearing somewhere that the Pi4 only has hardware decoding for H.265/HEVC, hardware decoding for H.264 and MPEG-2 are no longer supported, as the processor can now handle that in software.

#64

Hi tmm1...
Do you plan to merge your v4l2 hw scaler to ffmpeg mainline?. I think it would be quite useful having available a hw implementation working now that looks v4l2 drivers will be the new standard in linux.
Regards,

#65

Any idea what this is about? RPi 4 transcoding is not working for me.

2020/06/21 14:28:21.586324 [HLS] ffmpeg: file2-ip192.168.2.1:  [mpegts @ 0xfae810] Dropped corrupted packet (stream = 1)
2020/06/21 14:28:21.586406 [HLS] ffmpeg: file2-ip192.168.2.1:  [mpegts @ 0xfae810] Dropped corrupted packet (stream = 2)
2020/06/21 14:28:21.587729 [HLS] ffmpeg: file2-ip192.168.2.1:  [mpegts @ 0xfae810] stream 1 : no PTS found at end of file, duration not set
2020/06/21 14:28:21.587819 [HLS] ffmpeg: file2-ip192.168.2.1:  [mpegts @ 0xfae810] stream 2 : no PTS found at end of file, duration not set
2020/06/21 14:28:21.591105 [HLS] ffmpeg: file2-ip192.168.2.1:  [mpegts @ 0xfae810] Dropped corrupted packet (stream = 2)
2020/06/21 14:28:21.604267 [HLS] ffmpeg: file2-ip192.168.2.1:  [mpegts @ 0xfae810] Dropped corrupted packet (stream = 1)
2020/06/21 14:28:21.604342 [HLS] ffmpeg: file2-ip192.168.2.1:  [mpegts @ 0xfae810] Dropped corrupted packet (stream = 2)
2020/06/21 14:28:21.725134 [HLS] ffmpeg: file2-ip192.168.2.1:  [scale_v4l2m2m @ 0x10e1720] Could not find a valid device
2020/06/21 14:28:21.725278 [HLS] ffmpeg: file2-ip192.168.2.1:  [Parsed_scale_v4l2m2m_1 @ 0x10e1680] can't configure encoder
2020/06/21 14:28:21.725318 [HLS] ffmpeg: file2-ip192.168.2.1:  [Parsed_scale_v4l2m2m_1 @ 0x10e1680] Failed to configure output pad on Parsed_scale_v4l2m2m_1
2020/06/21 14:28:21.725341 [HLS] ffmpeg: file2-ip192.168.2.1:  [scale_v4l2m2m @ 0x10e1720] VIDIOC_STREAMOFF (null)
2020/06/21 14:28:21.725625 [HLS] ffmpeg: file2-ip192.168.2.1:      Last message repeated 1 times
2020/06/21 14:28:21.725704 [HLS] ffmpeg: file2-ip192.168.2.1:  Error reinitializing filters!
2020/06/21 14:28:21.725750 [HLS] ffmpeg: file2-ip192.168.2.1:  Failed to inject frame into filter network: Permission denied
2020/06/21 14:28:21.725773 [HLS] ffmpeg: file2-ip192.168.2.1:  Error while processing the decoded data for stream #0:0
2020/06/21 14:28:21.754477 [ENC] Encoder stopped for Young Sheldon S03E03 2019-10-10 An Entrepreneurialist and a Swat on the Bottom 2020-06-18-1959.mpg in /mnt/media/dvr/Streaming/file2-ip192.168.2.1-095906896/encoder-66-996543236 after starting from 66 without encoding any segments
2020/06/21 14:28:21.763631 [ENC] Starting encoder for Young Sheldon S03E03 2019-10-10 An Entrepreneurialist and a Swat on the Bottom 2020-06-18-1959.mpg in /mnt/media/dvr/Streaming/file2-ip192.168.2.1-095906896/encoder-66-234484371 at 66 (77.177100) (encoder=h264_omx, resolution=720, deinterlacer=blend, bitrate=4000 segment_size=0.01)
#66

Probably need firmware update with sudo rpi-update

#67

Have run rpi-update multiple times, and through a few updates of Channels DVR. Same bug. Any more suggestions?

#68

What's uname -a and v4l2-status say

#69

4 posts were split to a new topic: Commercial detection issue on RPI4

#70

Sorry this is the right command:

v4l2-ctl --list-devices

Can you also click Support > Submit Diagnostics at the top right of the web UI

#71

Sorry for the slow response. Here you are...

m@raspberrypi : ~ $ sudo v4l2-ctl --list-devices
[sudo] password for m:
bcm2835-codec-decode (platform:bcm2835-codec):
/dev/video10
/dev/video11
/dev/video12

bcm2835-isp (platform:bcm2835-isp):
/dev/video13
/dev/video14
/dev/video15
/dev/video16

Cannot open device /dev/video0, exiting.

m@raspberrypi : ~ $ uname -a
Linux raspberrypi 5.4.61-v7l+ #1339 SMP Tue Sep 1 18:51:27 BST 2020 armv7l GNU/Linux

#72

The video scaler is part of the ISP device iirc. That's showing up on the list, though I'm not sure why there are four of them.

There is a v4l2 command to list details about each of the devices including codecs and operations supported. Maybe this:

v4l2-ctl -D -d 11