New remuxer

chDVRuser · July 14, 2019, 7:57pm

I can confirm it falls back to the old ffmpeg remux for cable H.264 recordings.
I tried recording a few new TVE H.264 recordings and noticed something unusual when playing back in the web UI.
It appears to use the new remuxer when playing back in the web UI.
When the playback reaches the last few seconds of the recording the player shows the spinning buffer circle and never finishes the playback.

Here's the log from the last one I played.

2019/07/14 11:15:23 [DVR] Waiting 14m36.587735128s until next job 1563129000-ch6110 How It's Made
2019/07/14 11:30:00 [DVR] Starting job 1563129000-ch6110 How It's Made on ch=[6110]
2019/07/14 11:30:05 [TNR] Opened connection to TVE-Comcast_SSO for ch6110 SCIENCE
2019/07/14 11:30:05 [DVR] Recording for job 1563129000-ch6110 from TVE-Comcast_SSO ch6110 into "TV/How It's Made/How It's Made S28E03 2016-09-26 Angle grinders Berry baskets Omnidirectional speakers 2019-07-14-1130.mpg" for 29m59.991010318s
2019/07/14 11:30:05 [DVR] Generating m3u8 for job 1563129000-ch6110
2019/07/14 12:00:05 [TNR] Closed connection to TVE-Comcast_SSO for ch6110 SCIENCE
2019/07/14 12:00:05 [MTS] Statistics for "TV/How It's Made/How It's Made S28E03 2016-09-26 Angle grinders Berry baskets Omnidirectional speakers 2019-07-14-1130.mpg": skipped=0 unhandled_packets=0 discontinuity_detected=36 transport_errors=0 invalid_pts=0 invalid_dts=0 saw_pcr=false saw_pmt=true highest_pts=1811.525911
2019/07/14 12:00:05 [DVR] Finished job 1563129000-ch6110 How It's Made
2019/07/14 12:00:05 [DVR] Processing file-1510: TV/How It's Made/How It's Made S28E03 2016-09-26 Angle grinders Berry baskets Omnidirectional speakers 2019-07-14-1130.mpg
2019/07/14 12:00:06 [DVR] Running commercial detection on file 1510 (TV/How It's Made/How It's Made S28E03 2016-09-26 Angle grinders Berry baskets Omnidirectional speakers 2019-07-14-1130.mpg)
2019/07/14 12:07:05 [DVR] Commercial detection finished with 8 markers.
[HLS] Couldn't get PAT/PMT: unexpected EOF[HLS] Couldn't get PAT/PMT: unexpected EOF[HLS] Couldn't get PAT/PMT: unexpected EOF[HLS] Couldn't get PAT/PMT: unexpected EOF[HLS] Couldn't get PAT/PMT: unexpected EOF[HLS] Couldn't get PAT/PMT: unexpected EOF[HLS] Couldn't get PAT/PMT: unexpected EOF[HLS] Couldn't get PAT/PMT: unexpected EOF[HLS] Couldn't get PAT/PMT: unexpected EOF[HLS] Couldn't get PAT/PMT: unexpected EOF[HLS] Couldn't get PAT/PMT: unexpected EOF

The last line with repeating '[HLS] Couldn't get PAT/PMT: unexpected EOF' was from the playback when it started buffering 4 secs from the end of the recording, almost like it didn't play that last hls segment of 4.004 secs before the end.
#EXT-X-BYTERANGE:2639668@1146468556
#EXTINF:4.004,
stream.mpg
#EXT-X-ENDLIST

chDVRuser · July 14, 2019, 9:07pm

web UI player is requesting the last segment of the playlist
#EXTINF:4.004,
stream.m3u8/stream762.ts
#EXT-X-ENDLIST

Channels DVR returns a 500 error

Request URL:https://192.168.1.4:8089/dvr/files/1510/hls/stream.m3u8/stream762.ts
Request method:GET
Remote address:192.168.1.4:8089
Status code: 500 Internal Server Error
Version:HTTP/1.1

Result is player shows buffering and this error repeats in the log
[HLS] Couldn't get PAT/PMT: unexpected EOF

eric · July 14, 2019, 9:26pm

Thanks for reporting this. I’ll look into it and let you know when I’ve pushed a fix.

chDVRuser · July 16, 2019, 12:20am

Not sure if this applies to your code?, but the new streaming index files created for recordings (stream.m3u8) are using the tag EXT-X-BYTERANGE, but specifying EXT-X-VERSION:3 instead of 4.

#EXTM3U
#EXT-X-VERSION:3
#EXT-X-MEDIA-SEQUENCE:0
#EXT-X-TARGETDURATION:5
#EXT-X-BYTERANGE:2718856@0
#EXTINF:4.104,
stream.mpg
...

https://datatracker.ietf.org/doc/draft-pantos-hls-rfc8216bis/

4.4.2.2 - EXT-X-BYTERANGE
Use of the EXT-X-BYTERANGE tag REQUIRES a compatibility version number of 4 or greater.

7 - Protocol Version Compatibility
A Media Playlist MUST indicate an EXT-X-VERSION of 4 or higher if it contains:
o The EXT-X-BYTERANGE tag.
o The EXT-X-I-FRAMES-ONLY tag.

eric · July 16, 2019, 1:48am

Have you run into any issues with this? We haven't intended to be exposing this file as-is externally and are just using it as our internal index so it hasn't been an issue for us.

I'll be pushing a new build that sets it to version 6 to be more consistent with the things that we are including in it.

chDVRuser · July 16, 2019, 2:18am

Not at all. That's why I mentioned I wasn't sure if it applied to your code, since it's not exposed externally for anyone to use.

I did want to add that the issue I'm seeing seems to be sporadic. Not all TVE recordings are failing, but the ones that fail always fail, the ones that don't, don't. Will delete them all (they were only for testing) and test some more TVE recordings, but the last two TVE recordings I made worked fine (could be because I was on ver 2019.07.15.2123, now 2019.07.16.0113?).

chDVRuser · July 17, 2019, 2:45am

Still having issues with (some) TVE recordings playing back (and remuxing) in the web UI.

I was only making TVE recordings to help test the new remuxer, will be deleting all of them

Giving up on Beta testing this for now.

Too many new versions being released and don't know what the changes are or what to test for.

I record with one version, a new version comes out and I play the previous recording with that but have no idea what I'm testing for.

Would like to help, but will quit wasting my time for now as you seem to have a handle on it.

tmm1 · July 17, 2019, 4:50pm

Sorry for the confusion. There are a lot of moving parts and we have been scrambling to update the dvr and apps to work well together when remuxing recordings.

The current status is:

all h264 and mpeg2 recording with a streaming index are streamed directly to the Channels app with the new remuxer
all TVE h264 recordings with a streaming index are streamed directly to both Channels app and web player using the new remuxer
all other h264 recordings (i.e from cable with ac3 audio, or missing streaming index) are transcoded with ffmpeg for web playback (to convert audio to aac and to add video keyframes every second)

We are confident (1) is working well now, and the dvr Activity says "Streaming file.mpg" when this kicks in.

We are also confident (3) is working as expected, and the web player will always show "Transcoder Running" below it in this case.

For (2) the web player will say "Streaming directly" underneath. This case has not been thoroughly tested, but it should be working well and we are not aware of any remaining issues.

If you're having problems with (2) please let us know what's happening. There's no need to make new recordings as the recording code and index generator have both stabilized. We will post something if either of those are updated.

chDVRuser · July 18, 2019, 11:28pm

Thanks!
Felt like I was just spinning my wheels and wasting my time testing things while not knowing what was being worked on and what was fixed.
Understand when things move so fast you don't have time to keep us updated.
Keep up the great work.

carterbunch · July 19, 2019, 5:58pm

I am not seeing this show in the logs to verify that it working.

For 2 does the web player is this only for recorded and not Live TV?

tmm1 · July 19, 2019, 6:01pm

"Streaming file.mpg" will show up in the Activity section of the DVR web UI.

Yes, only for recordings. There have been no changes to Live TV remuxing or transcoding.

carterbunch · July 19, 2019, 6:24pm

So I have this recording from OTA source and I am expecting case 1 for the iOS Channes app and Case 2 for web app, but I am seeing case 3 for web app.

fullsizeoutput_b1c3 fullsizeoutput_b1c4 fullsizeoutput_b1c2 fullsizeoutput_b1bf

I am seeing the same results for this movie from TVE source

fullsizeoutput_b1c6 fullsizeoutput_b1c7 fullsizeoutput_b1c5 fullsizeoutput_b1c8

chDVRuser · July 20, 2019, 9:02pm

Should be #3 for mpeg-2 w/AC-3 audio. Has to be transcoded since a web browser can't play those codecs.

That shouldn't transcode unless you have some setting causing it to.

tmm1 · July 20, 2019, 9:04pm

I was mistaken here. The app will only use the remuxer when not in Original mode, which is generally only when you're remote.

But our plan is to make it use the new remuxer at home as well, since seeks are much faster and everything feels less laggy.

carterbunch · July 28, 2019, 12:12pm

Is there a particular quality setting I need on the web player to trigger this? Right now I have 720p @ 3mbps and I get get it to stream using the remuxer.

carterbunch · July 30, 2019, 2:00pm

What bit rate is sent to client when using the new remuxer? With the transcoder you can specify bitrate. Are there options to do that with the new remuxer?

carterbunch · July 30, 2019, 2:02pm

This recording is doing direct streaming correctly now local or remote.

racameron · July 30, 2019, 2:50pm

By definition, remuxing would not change the bitrate. It simply takes whatever streams were in the MPEG Transport Stream or HTTP Live Stream and repackages them into an HTTP Live Stream with indexing and segments that better suit a Channels client.

If changes were to be made to the bitrate, that would be transcoding because the video would have to be decoded and re-encoded with a new bitrate.