Tuning my setup to remove stalls, pixelation - TVHeadend, UK Freesat, Channels, Sat>IP

Initially when I moved from TVHeadend / OSMC to Channels DVR with Fire TVs, everything ran more or less flawlessly. However, as time went on, bits of hardware changed, software and kernels changed, the odd stall and periods of pixelation began creeping into my experience.

Eventually, it got to the point where I got the bit between my teeth and went hunting for the cause.

What became apparent, and is often the case, is that there was not one single cause - whilst, yes, one or two aspects certainly had more impact than others, there was not one cause.

My deteriorated experience was a cumulation of various changes introduced into my environment.... by me.

So, I figured I would document my experiences here in case it helps some one else. I believe I am now (back) at the point where the experience is more or less flawless.

My setup is as follows:

  • Ubuntu 20.04, with the HWE kernel (linux-image-generic-hwe-20.04, I believe this provides additional Realtek NIC enhancements/fixed)
  • Gigabyte Brix, i7-3537U, 16GB RAM
  • Toshiba USB 3.0 external drive
  • Megasat Sat>IP Server 3 MEG-8000
  • TVHeadend, custom/minimal build
  • Channels DVR
  • Mainly Fire TV Cubes, 1 Fire Stick 4K
  • Fully Unifi/Ubiquiti network
  • CAT6A everywhere
  • No wifi as part of the network path for this setup

Some of this will be TVHeadend specific, some will be specific to Sat>IP, some specific to my Sat>IP server.

Much of this will not apply and should not be followed if you're using a Pi, a virtual environment, or running Channels on a NAS.

In the early days I did run everything in a virtual environment, but my life is now infinitely better since running it on reasonably powerful, dedicated hardware - i.e I get moaned at less when 'fiddling' with things on the virtual environment, that ends up impacting media/TV.

I'm going to break this down into a few main areas:

  • TVHeadend
  • OS tuning (sysctl.conf, power management)
  • Network Infrastructure

TVHeadend
I essentially use TVHeadend as the missing UI for my Sat>IP server, it really doesn't do much apart from interfacing with the Sat>IP server and then relaying via the M3U export to Channels. There are LOTS of aspects that I simply do no need, so I remove these from the build.

I'm not going to redocument the building process, you can find it below:

https://tvheadend.org/projects/tvheadend/wiki/Packaging

My stripped down build options, essentially leaving Sat>IP client enabled and disabling more or less everything else I don't need.

AUTOBUILD_CONFIGURE_EXTRA="--python=/usr/bin/python3 --disable-pcloud_cache --disable-ddci --disable-tvhcsa --disable-epoll --disable-ffmpeg_static --disable-libav --disable-libx264_static --enable-libx264 --disable-libx265_static --enable-libx265 --disable-libvpx_static --disable-libtheora_static --disable-libvorbis_static --disable-libfdkaac_static --disable-libopus_static --disable-nvenc --disable-hdhomerun_static --disable-hdhomerun_client --disable-cardclient --disable-cwc --disable-cccam --disable-capmt --disable-constcw --disable-linuxdvb --disable-satip_server --disable-iptv --disable-dvbscan --disable-timeshift --disable-trace --disable-avahi --disable-inotify --disable-ccache --disable-pie" ./Autobuild.sh -t focal-amd64

TVHeadend brings a couple of benefits for me:

  • Channel tuning/change is faster, than when using Channels DVR -> Sat>IP with an M3U
  • It provides a UI to create the M3U, adding channels, moving them around, customising channels numbers, etc
  • I can record more channels as once, due to the Mux sharing. For example, BBC One HD and BBC Two HD are broadcast on the same Mux, so I can record both of these channels at the same time, only using 1 input on my Sat>IP box
  • I can see things like 'Uncorrected errors' and 'Continuity Errors', i.e basically when bits of the data/video stream go missing, this could be ANYWHERE from the TV broadcast/encoding end, right down to the interface/software on my Server, unfortunately.

TVHeadend settings I found that helped:

Configuration -> DVB Inputs -> TV Adapters -> Disable Fast Input Switch, Disable Full Mux RX mode

OS Tuning
Some of you will agree with my sysctl.conf settings, some of you won't - tuning is appropriate, not overly aggressive, for my hardware and network which is all cabled.

With my Sat>IP server I can record from 8 Mux at once, due to the way TVHeadend can also share a Mux it's possible to record from more than 8 channels at once, providing there are no more than 8 Mux required.

I tested up to 12 concurrent streams, for this I needed to increase the buffers (UDP especially) to prevent drops. You can check for UDP drops with something like 'netstat -suna'. So, my sysctl.conf ends up as follows:

/etc/sysctl.conf

# Disable IPv6
net.ipv6.conf.all.disable_ipv6 = 1

# Reboot 20 seconds after a kernel panic
kernel.panic = 20

# Even out the writes, to prevent periods of heavy IO
vm.dirty_background_bytes=16777216
vm.dirty_bytes=33554432

# We don't want to swap
vm.swappiness=0

# Tune the buffers (UDP and TCP) and increase default starting point for a gigabit network
net.core.rmem_default = 16777216
net.core.wmem_default = 16777216
net.core.rmem_max = 33554432
net.core.wmem_max = 33554432
net.ipv4.udp_mem = 8388608 12582912 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 87380 16777216
net.ipv4.udp_rmem_min = 32768
net.ipv4.udp_wmem_min = 32768
net.ipv4.tcp_fastopen = 3
net.ipv4.tcp_slow_start_after_idle = 0

# Not strictly necessary, but is part of my 'default' config tuning, assymetric routing is not a problem in this environment
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.all.rp_filter = 1

# <3 TCP BBR / FQ, this is another topic entirely
net.ipv4.tcp_congestion_control=bbr
net.core.default_qdisc=fq

# Increase the backlog, max conns, file inotify limit these perhaps ARE overly aggressive for the environment but provides no detremental impact for me
net.core.netdev_max_backlog = 2500
net.core.somaxconn = 65000
fs.inotify.max_user_watches=524288

Power Management
This, aside from the network infrastructure, probably had the single biggest impact on my hunt for stalls and pixelation.

Essentially, when the CPU scales in frequency, I would often see an increase in 'Uncorrected errors' and 'Continuity Errors' on TVHeadend - packet loss, basically. The Sat>IP streams over UDP -> TVHeadend, so interruptions or dropped packets will not be resent by the network stack as would happen with TCP (unless built into the application, which it isn't).

Since, for me, flawless streaming was more important than power saving, I disabled Intel P State (so it reverts to ACPI based scaling, i.e the OS controls it rather than the chip), C State and set Idle to Poll.

/etc/default/grub

GRUB_CMDLINE_LINUX_DEFAULT="intel_idle.max_cstate=0 processor.max_cstate=0 idle=poll intel_pstate=disable"

Make sure to 'update-grub', reboot for changes.

I also set cpufrequtils governor to run in 'Performance' mode, basically don't decrease power to the CPU at all.

/etc/default/cpufrequtils

GOVERNOR="performance"

Some people on the TVheadend forums have also had good experiences with setting a minimum frequency of around 2.0Ghz, and leaving powersave mode enabled - to allow the CPU to scale up and down, within set limits - the problems in these cases seemed to occur when the CPU was allowed to drop below 1.3 - 2.0 Ghz, the drop in power seems to cause the ethernet interface to drop packets, impacting UDP particularly, causing stalls and pixelation.

Given that setting a minimum frequency of 2.0Ghz is more or less full power for this CPU, I chose to keep performance mode which is ACPI mode means don't scale.

This is a hardware / power saving issue, not a TVheadend/Channels issue. Although it would be good if Channels could alter the governor, based on what the setup is doing - i.e when recording/playing, set performance, when idle set back to powersave.

I'm not sure if there is anyway to run a script pre/post Channels doing 'something', anyone?

Network Infrastructure
I have the added benefit that my house is CAT6A cabled, everywhere. So from the Sat>IP server in the loft, to the rack under the stairs, to the Fire TVs, everything is cabled.

Whilst many people simply see 'no network problems' because you can stream Youtube, Netflix at 4K, etc, this is not really a like-for-like comparison - especially if you have devices streaming UDP over your network.

Along with dedicated hardware - i.e not a virtual machine that is contending for resources, or a NAS that is running download clients, TimeMachine backups, etc - cables-everywhere make things infinitely better than using wifi.

For around 5-6 years, I had 2 x TP Link switches for the rack core/aggregation, then some additional Netgear switches in the various rooms if I needed more ports, with 3 x Unifi AC Pros for wifi. This setup was actually pretty good, a few niggles, but overall pretty stable.

Then, I decided to swap the switching infrastructure for 'pretty shiny' Unifi/Ubquiti and I honestly spent many weeks wishing I hadn't, it caused me no end of hassles.

My experience of Unifi/Ubquiti is not especially favourable over the last year, although I believe I am now at the point where it is perhaps starting to become more favourable. Unifi/Unquiti often seem to change many, many more things than the release notes state - often it is just 'bug fixed'.

What kind of problems was a I seeing? You name it - UDP drops, multicast working and not working, devices rebooting, IGMP snooping completely dying between a x.x.1 incremental release, throughput issues on wifi with tagged VLANs. I spent considerable time testing with iPerf/PerfSonar, checking for jitter/drops/throughput and experimenting with different firmware releases.

For my setup, I have found the below to provide a flawless experience:

Switches - 5.43.23 (earlier 5.43.x and ALL 4.3.x releases were a disaster for me)
Wifi AC Pros - 5.43.24 (earlier 5.43.x and ALL 4.3.x releases were a disaster for me)

I did have some Flex Mini switches, whilst neat little devices and hard to argue with the price, I would consistently see periods/bursts of UDP packet loss - which when you've got one connected to a Sat>IP server, that streams over UDP, caused pixelation and stalls.

When the Flex mini was swapped out for a regular Flex (non mini), the problems ceased. So, they've been relegated to the 'toy cupboard'.

Final thoughts...
If the video feed/stream is 'dirty' on the way in, it will be dirty on the way out when you watch it.

After going through the above, I got to the point where I was happy and believed I had resolved my problems.....and then.....I began to see stalls and pixelation, I wanted to throw it all in the river :wink:

But, I noticed that it was only on 2 channels - Channel 4 HD and 4Seven.

As it turned out, these 2 channels were on the same Mux/frequency, either there was an encoding problem at Channel 4, or some form of satellite issue. In sheer desperation I began to search Twitter and the Sky forums and found other examples of similar problems. Once these were resolved either by Channel 4, or the satellite provider, everything went back to being flawless.

4 Likes

Looks like there were some changes in the 5.8 Linux kernel relating to power management and Intel P state - at least for older i7 CPUs it now defaults to passive mode for P state, along with the schedutil governor seems to have resolved my problems.

On Ubunu 20.04 running the 'linux-image-generic-hwe-20.04-edge' kernel, which is a 5.8 kernel, currently 5.8.0.44.50~20.04.30. So I'm now back to a scaling CPU and still buffering, stall, pixelation free. Happy days.

Changed /etc/default/grub back to:

GRUB_CMDLINE_LINUX_DEFAULT=""

Then:

update-grub

Then specifying schedutil for the governor in /etc/default/cpufrequtils:

GOVERNOR="schedutil"

/etc/init.d/cpufrequtils restart

Then reboot for the grub changes to be activated.

1 Like