New Option on Recording - Regenerate Streaming Index

[carterbunch]

I noticed this new option for recorded shows. What does it do?

[tmm1]

This is part of our new transcoder/remuxer. It's still in development but we will be releasing more details on how to try it out soon.

[tmm1]

Starting with v2019.07.11.2248 our new remuxer is enabled by default for recordings with an up-to-date streaming index. You can click "View Details" for a recording to see if it says "Up-to-date: Yes"

If not, you can use the "Regenerate" option to generate it. New recordings will automatically generate the index while recording, but older builds will need to be manually generated. (Future DVR version will automatically regenerate in the background once we're confident in the indexer.)

This is most helpful for TVE recordings which are in H264 already and were stalling during playback in the browser before. Now they will play correctly, and you'll notice much faster seeking and playback performance.

[racameron]

Is there an API endpoint we can use to initiate the generation of the index?

[tmm1]

Yes, PUT /dvr/files/XXX/m3u8

[racameron]

Awesome! Thanks!

For those interested in (re-)generating the M3U8s for all recordings, here's my oneliner:

for _f in /media/channels/Logs/comskip/*
do
  curl -sSXPUT http://127.0.0.1:8089/dvr/files/$(basename ${_f})/m3u8
done

(Of course, this will only do it for recordings that have had their commercials indexed; on my system that just so happens to be all of them. Also, don't forget to substitute the root of your recordings directory, as yours may not be /media/channels/ like mine is.)

[tmm1]

If you do decide to run over everything, keep an eye out in the Log for any errors like this:

2019/07/11 20:10:50 [M3U8] m3u8 generation for /dvr/TV/The Rifleman/The Rifleman S01E08 1958-11-18 The Safe Guard 2019-07-11-1500.mpg failed with MPEG-TS discontinuity: PTS jump was too far: 204963823039951.750000

and report them to us. Once the new indexer is considered stable, we plan to automatically run over everyone's recordings in the background so you don't have to run any scripts.

[LAdams]

I just tried this on an older build and it worked perfectly. Great addition for me since I only watch TV on my Windows 10 computer using the Web UI.

Thank you.

[eric]

If you have jq installed you can use this trick:

for id in $(curl -qs http://localhost:8089/dvr/files | jq -r '.[].ID'); do
  if [ "true" != "$(curl -sq http://localhost:8089/dvr/files/$id/mediainfo.json | jq -r '.m3u8_up_to_date')" ]; then
    echo "Regenerating $id"
    curl -sSXPUT "http://localhost:8089/dvr/files/$id/m3u8"
    echo
  fi
done

It'll grab all of the files on your DVR and only regenerate the ones that need to be.

[racameron]

Much better! (I'm not at home, so I didn't really have time to think something up better. And I indeed have jq.)

[AeroR1]

This new remuxer is great! it fixed the crappy stream i was getting from SNY. Hopefully this new transcoder that is due to come will fix my hardware transcoding issue. great work!

[carterbunch]

If you do decide to run over everything, keep an eye out in the Log for any errors like this:

2019/07/11 20:10:50 [M3U8] m3u8 generation for /dvr/TV/The Rifleman/The Rifleman S01E08 1958-11-18 The Safe Guard 2019-07-11-1500.mpg failed with MPEG-TS discontinuity: PTS jump was too far: 204963823039951.750000

I don't get those errors, but I am seeing a lot of these types of errors:

2019/07/15 11:18:04 [M3U8] m3u8 generation for /Volumes/ChannelsDVR/recordings/TV/Today/Today 2019-07-15 2019-07-15-0700.mpg failed with No Keyframes were found in the stream
2019/07/15 11:19:19 [M3U8] m3u8 generation for /Volumes/ChannelsDVR/recordings/TV/Today/Today 2019-07-15 2019-07-15-0700.mpg failed with No Keyframes were found in the stream
2019/07/15 11:19:19 [M3U8] Regenerating m3u8 for /Volumes/ChannelsDVR/recordings/TV/Jackie Robinson/Jackie Robinson S01E02 2016-04-12 Part Two 2019-07-14-2300.mpg
2019/07/15 11:20:37 [M3U8] m3u8 generation for /Volumes/ChannelsDVR/recordings/TV/Jackie Robinson/Jackie Robinson S01E02 2016-04-12 Part Two 2019-07-14-2300.mpg failed with No Keyframes were found in the stream

Server Version: 2019.07.15.0203

[tmm1]

Huh weird. Do those recording play back normally? Can you upload one of them.

[carterbunch]

I believe they playback normally. Since they are pretty large where would you suggest I upload them?

Here is the output of the regenerate command:

Regenerating 5137
{"ID":"5137","JobID":"1563152400-ch8.1","RuleID":"","GroupID":"11114803","Path":"TV/Grantchester on Masterpiece/Grantchester on Masterpiece S04E00 2019-07-14 2019-07-14-2100.mpg","Checksum":"","CreatedAt":1563152400,"Watched":false,"Deleted":false,"PlaybackTime":0,"Duration":7200.832456,"Commercials":[0,5.81,38.91,146.61,6718.81,6749.61,6795.52,7199.83],"Processed":true,"Airing":{"Source":"tms","Channel":"8.1","OriginalDate":"2019-07-14","Time":1563152400,"Duration":7200,"Title":"Grantchester on Masterpiece","Summary":"Sidney is swept up in the civil rights movement after a pastor and his daughter arrive in Cambridge.","Image":"http://fanc.tmsimg.com/assets/p11114803_b_h6_ap.jpg","Categories":["Episode","Series"],"Genres":["Crime drama","Mystery","Entertainment"],"Tags":["Season Premiere","New","HD","DVS","CC"],"SeriesID":"11114803","ProgramID":"EP025199990022","TeamIDs":null,"SeasonNumber":4,"EpisodeNumber":0,"Raw":{"startTime":"2019-07-15T01:00Z","endTime":"2019-07-15T03:00Z","duration":120,"channels":["8.1"],"stationId":"59210","qualifiers":["CC","DVS","HD 1080i","HDTV","New","Season Premiere","Stereo"],"ratings":[{"body":"USA Parental Rating","code":"TV14"}],"program":{"tmsId":"EP025199990022","rootId":"16910724","seriesId":"11114803","subType":"Series","title":"Grantchester on Masterpiece","titleLang":"en","seasonNum":4,"releaseYear":2019,"releaseDate":"2019-01-11","origAirDate":"2019-01-11","entityType":"Episode","descriptionLang":"en","shortDescription":"Sidney is swept up in the civil rights movement after a pastor and his daughter arrive in Cambridge.","longDescription":"Sidney is swept up in the civil rights movement after a pastor and his daughter arrive in Cambridge, and a murder sees racial tensions spike; Sidney struggles to remember a murdered woman's final words.","topCast":["James Norton","Robson Green","Tessa Peake-Jones"],"directors":["Tim Fywell"],"genres":["Crime drama","Mystery","Entertainment"],"preferredImage":{"uri":"http://fanc.tmsimg.com/assets/p11114803_b_h6_ap.jpg","height":"540","width":"720","primary":"true","category":"Banner-L1","text":"yes","tier":"Series"}}}},"PlayedAt":0,"UpdatedAt":1563193168044,"HighestPTS":0,"CommercialsAligned":true,"CommercialsEdited":true,"CommercialsVerified":false}

[tmm1]

You can upload to Firefox Send, Google Drive, Microsoft OneDrive, Dropbox or whatever is most convenient.

[carterbunch]

Here is the link to the file. file_link

Let me know after you have downloaded so I can remove. Thanks.

[tmm1]

Thanks. I am downloading now.

I poked at the beginning of the file that downloaded and it looks like an mp4 file. Are you processing your recordings somehow?

[carterbunch]

Yes. I use ffmpeg to wrap the mpeg in an mp4 container to add commercial markers so plex and see the chapters and allow you to skip in plex.

[carterbunch]

Using this script

Script to repackage recordings for Plex commercial skipping

I want to share a bash script I wrote to repackage TV recordings for Plex so you can utilize the skip commercials on the Plex side. I have only tested this on Mac environment and it is very rough script with minimal error checking. I also read, but can't find that Channels devs are working on post record webhook/trigger? This would be great to allow this to run after a recording has finished. Right now the script is run against the comskip directory and processes everything there. It tries …

[tmm1]

Gotcha, okay. This is normal then. The new remuxer streaming index will not work on mp4 and other re-processed files.