BETA: New in-memory Tuner Sharing Buffer

[eric]

We have just rolled out a new DVR build to test out a new Tuner Sharing Buffer to improve visibility and problems that sometimes occur with slow clients.

DVR Pre-Release Notes

v2022.08.09.1854 • NEW: Experimental Tuner Sharing Buffer to resolve issues with slow clients

Currently, it is possible for a single slow client to cause other clients (or recordings) tuned into the same channel to stall and eventually fail. When we have moved to the new Tuner Sharing Buffer we expect these issues to go away.

We've also added additional stats in the HDHR stats view to expose the current size of the buffer to help understand if slow clients are causing issues. They are labeled as buf= (percentage of buffer filled) and drop= (percentage of bytes dropped).

Please note: This new buffer does not provide any new capabilities. It is purely to fix bugs and behaviors in the existing mechanism.

Please note: This does not do anything to help with retroactive recording.

Behavior of new buffer

Previously a single very slow client could cause all other clients to stall (and recordings to potentially fail). With the new Streaming Buffer, slow clients will have data dropped but the other clients will continue on unimpeded. In the latest client betas, the client will also see the "Network Tuner Error" notification in the top right corner when these drops are occurring.

The difference in the old and new buffer will be most apparent when having (or creating) very slow clients. The main goal is to prevent situations like this from causing other clients to stall:

Stream.mpeg M3U Buffering Bug

Found my bug and can duplicate the Channels streaming bug on demand. Start watching an M3U channel on a TVoS client and then: curl "http://dvr.lxc.lan:8089/devices/ANY/channels/7002/stream.mpg" >/dev/null Once curl has been running for a minute or so (establish data flow), control-Z to suspend it. Within ten minutes, channels will overflow its ring buffer and the well-behaved TVoS client will fail with "Connection Lost". When the misbehaving client stops accepting data (but its connection rema…

Enabling new buffer for testing

IMPORTANT: Please do not enable this feature if you would be sad if your future recordings were corrupted or ruined

We have done every testing and code review we can before shipping this, but with code that is this important to the recording process, it is possible that there are bugs.

We are looking for brave souls and folks who like testing to enable this and let us know what their experience is.

To enable the new Tuner Sharing Buffer, scroll to the bottom of the DVR Settings page and enable the setting New Streaming Buffer:

Please leave your comments on this thread if you see any issues.

[Edwin_Perez]

I guess this will only work when tuner sharing is enabled. Is this different from HLS ?

[tmm1]

Yes this is a new implementation of Tuner Sharing

It is not related to HLS

[Edwin_Perez]

Thanks will turn it on not worried if I have bad recordings this eve. Will report back.

[Edwin_Perez]

One thing I notice that if you are watching a channel local and the same channel remote (LTE) it will use a second tuner on that device ? .... is this normal?

[tmm1]

The behavior should be the same between both modes, so if its not sharing a tuner when it should be that sounds like a bug.

[Edwin_Perez]

I see what the problem was one of my clients did not connect to the DVR so it just used the tuner directly working ok now sorry about the false alarm.

[Edwin_Perez]

The buf is always 0 ... is that expected when it is running smoothly.

Watching ch744 CNNHD from Edwin's 2nd Fire TV Cube: strength=85% quality=100% symbol=100% rate=3.9Mb/sec buf=0%
Watching ch744 from 72.106.11.37 (Remux Running: 1m35s @ 1.02x (59.90fps)): strength=86% quality=100% symbol=100% rate=3.7Mb/sec buf=0%

[eric]

Yes, being empty is the preferred state. It will only show above 0 if there is something wrong going on.

[Edwin_Perez]

All recordings were fine no problem.

[dkrach]

One of my older AppleTVs that routinely stopped streaming after 10-15 minutes worked flawlessly last night. Will keep an eye on it, but so far this seems like an improvement.

[eric]

@dkrach did you have tuner sharing enabled? Did you enable the New Streaming Buffer?

[dkrach]

Yes, I have server side setup for Tuner sharing for all devices. After updating yesterday to the latest DVR pre-release I enabled the new streaming buffer. Just a single evening of use so far, but this particular older appleTV has always given me trouble with streaming stopping after a watching for a while.

[dkrach]

I just saw this thread, my issue was more similar to what was referred to here, but applied to all sources, local HDHRs and IP based streams. IP streams do no reconnect automatically when a connection is lost - #14 by eric

[chDVRuser]

Just noticed this during a recording and I have the Experimental New Streaming Buffer unchecked.

Is this the new normal?

It logged this when the recording finished

[SNR] Buffer statistics for "TV/Star Trek Voyager/Star Trek Voyager S01E03 1995-01-30 Time and Again 2022-08-12-1958.mpg": buf=0% drop=0%

[eric]

Yes, this is the new normal. We are now exposing the buffer usage for all streams.

Without the New Streaming Buffer enabled the drop= statistic will always be 0% (the old buffer is unable to drop) and the buf= percentage will be the same for all clients watching the same channel.

[peterkozmd]

Sorry if off topic but can someone link me to how to sign up for beta access. Thank you

[eric]

Open the DVR in your browser and hit the down arrow and select Pre-Release

[chDVRuser]

@eric

Without the New Streaming Buffer enabled the drop= statistic will always be 0% (the old buffer is unable to drop) and the buf= percentage will be the same for all clients watching the same channel.

So it appears the DVR record engine is a client?

[SNR] Buffer statistics for "TV/Magnolia Table With Joanna Gaines/Magnolia Table With Joanna Gaines S04E04 Favorite Sides 2022-08-14-0957.mpg": buf=0%-1% drop=0%

I think the buf=0%-1% was caused by Channels DVR re-authing the channel to record (which happens every 8 days for this Magnolia channel).

2022/08/14 09:57:00.000806 [DVR] Starting job 1660496220-12 Magnolia Table With Joanna Gaines on ch=[6108]
2022/08/14 09:57:28.797259 [TVE] action=fill_form u=USERNAME
2022/08/14 09:57:38.911505 [TVE] action=scienceauth done=true
2022/08/14 09:57:38.911552 [TVE] action=authed
2022/08/14 09:57:41.234409 [TVE] stream timestamps: diy: start_at=2022-08-14T09:56:41-07:00 current_at=2022-08-14T09:57:01-07:00 end_at=2022-08-14T09:57:09-07:00
2022/08/14 09:57:41.234598 [TNR] Opened connection to TVE-Comcast_SSO for ch6108 DIY
2022/08/14 09:57:41.267351 [DVR] Recording for job 1660496220-12 from TVE-Comcast_SSO ch6108 into "TV/Magnolia Table With Joanna Gaines/Magnolia Table With Joanna Gaines S04E04 Favorite Sides 2022-08-14-0957.mpg" for 34m59.999087284s
2022/08/14 09:57:41.616220 [IDX] Generating video index for job 1660496220-12
2022/08/14 10:32:00.001037 [SNR] Buffer statistics for "TV/Magnolia Table With Joanna Gaines/Magnolia Table With Joanna Gaines S04E04 Favorite Sides 2022-08-14-0957.mpg": buf=0%-1% drop=0%