Corrupted time/duration with latest MPEG-TS Rewriter (server v2023.02.19.2355)

mjitkop · February 20, 2023, 7:21pm

As mentioned in EXPERIMENTAL: New MPEG-TS Rewriter for HLS (TVE/M3U) - #68 by mjitkop

I'm guessing the bit rate is a simple calculation size/duration.

Relevant log entries:

2023/02/20 00:59:52.336688 [M3U] stream timestamps: hot-bench: start_at=2023-02-20T00:59:51-05:00 end_at=2023-02-20T01:00:11-05:00 live_delay=15s
2023/02/20 00:59:52.336830 [TNR] Opened connection to M3U-PlutoTV for ch11125 Hot Bench
2023/02/20 00:59:52.337397 [DVR] Recording for job 1676872790-194 from M3U-PlutoTV ch11125 into "TV\Hot Bench\Hot Bench S01E177 2014-01-01 Episode 177 2023-02-20-0059.mpg" for 31m9.9839419s
2023/02/20 00:59:52.452072 [IDX] Generating video index for job 1676872790-194

2023/02/20 01:28:30.855324 [SNR] Buffer statistics for "TV\Hot Bench\Hot Bench S01E177 2014-01-01 Episode 177 2023-02-20-0059.mpg": buf=0% drop=0%

2023/02/20 01:28:46.031526 [TNR] Closed connection to M3U-PlutoTV for ch11125 Hot Bench
2023/02/20 01:28:46.031526 [DVR] Error running job 1676872790-194 Hot Bench: The Channels DVR Server had a problem. Press play to try again. (Transcoder Reset): Playlist skipped to a higher sequence (356 -> 378): https://service-stitcher-ipv4.clusters.pluto.tv/stitch/hls/channel/6036e6e7ac69c400072afca2/3063648/playlist.m3u8?terminate=false&sid=15f49efd-2f01-4218-a05c-b0d038da9e54&deviceDNT=0&deviceModel=web&deviceVersion=unknown&includeExtendedEvents=false&serverSideAds=true&appName=web&deviceId=c954cb8f-b0e3-11ed-b04c-70b5e8567a24&appVersion=unknown&deviceType=web&deviceMake=Chrome
2023/02/20 01:28:47.050387 [DVR] Starting job 1676872790-194 Hot Bench on ch=[11125]

2023/02/20 01:28:49.307091 [M3U] stream timestamps: hot-bench: start_at=2023-02-20T01:28:44-05:00 end_at=2023-02-20T01:29:04-05:00 live_delay=15s
2023/02/20 01:28:49.307091 [TNR] Opened connection to M3U-PlutoTV for ch11125 Hot Bench
2023/02/20 01:28:49.307686 [DVR] Recording for job 1676872790-194 from M3U-PlutoTV ch11125 into "TV\Hot Bench\Hot Bench S01E177 2014-01-01 Episode 177 2023-02-20-0059.mpg" for 2m12.9496121s
2023/02/20 01:31:15.020092 [TNR] Closed connection to M3U-PlutoTV for ch11125 Hot Bench
2023/02/20 01:31:15.062161 [SNR] Buffer statistics for "TV\Hot Bench\Hot Bench S01E177 2014-01-01 Episode 177 2023-02-20-0059.mpg": buf=0% drop=0%
2023/02/20 01:31:15.062673 [DVR] Finished job 1676872790-194 Hot Bench

2023/02/20 01:31:15.365231 [DVR] Processing file-18340: TV\Hot Bench\Hot Bench S01E177 2014-01-01 Episode 177 2023-02-20-0059.mpg
2023/02/20 01:31:15.705767 [IDX] Generating video index for file-18340: TV\Hot Bench\Hot Bench S01E177 2014-01-01 Episode 177 2023-02-20-0059.mpg
2023/02/20 01:31:15.720979 [MTS] Rewriting video timestamps for file-18340: Hot Bench S01E177 2014-01-01 Episode 177 2023-02-20-0059.mpg
2023/02/20 01:31:17.868158 [MTS] Statistics for #18340 "Hot Bench S01E177 2014-01-01 Episode 177 2023-02-20-0059.mpg": skipped=0 unhandled_packets=0 discontinuity_detected=1 transport_errors=0 invalid_pts=0 invalid_dts=0 saw_pcr=true saw_pmt=true highest_pts=1915.355556
2023/02/20 01:31:18.200671 [ERR] Generating video index for file-18340 failed: replace D:\DVR\TV\Hot Bench\.Hot Bench S01E177 2014-01-01 Episode 177 2023-02-20-0059.mpg.1103564497 D:\DVR\TV\Hot Bench\Hot Bench S01E177 2014-01-01 Episode 177 2023-02-20-0059.mpg: Access is denied.
2023/02/20 01:31:18.515210 [IDX] Finished video index generation for file-18340 in 2s

Logs have been submitted as 8ce802e9-daa5-40f9-bc20-2517feb60dfb .

I can see in VideoRedo the moment the time becomes corrupted:

I just noticed in Windows Explorer that there is a weird file associated with it:

I tried to open it with Notepad++ and it's all binary data.

Media Info:

General
ID                                       : 0 (0x0)
Complete name                            : D:\DVR\TV\Hot Bench\Hot Bench S01E177 2014-01-01 Episode 177 2023-02-20-0059.mpg
Format                                   : MPEG-TS
File size                                : 541 MiB
Duration                                 : 26 h 4 min
Overall bit rate mode                    : Variable
Overall bit rate                         : 48.3 kb/s
FileExtension_Invalid                    : ts m2t m2s m4t m4s tmf ts tp trp ty

Video
ID                                       : 256 (0x100)
Menu ID                                  : 1 (0x1)
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile                           : [email protected]
Format settings                          : CABAC / 4 Ref Frames
Format settings, CABAC                   : Yes
Format settings, Reference frames        : 4 frames
Codec ID                                 : 27
Duration                                 : 26 h 4 min
Nominal bit rate                         : 2 400 kb/s
Width                                    : 1 280 pixels
Height                                   : 720 pixels
Display aspect ratio                     : 16:9
Frame rate mode                          : Variable
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Progressive
Writing library                          : x264 core 157 r2960 ae4b56b
Encoding settings                        : cabac=1 / ref=3 / deblock=1:0:0 / analyse=0x3:0x113 / me=hex / subme=7 / psy=1 / psy_rd=1.00:0.00 / mixed_ref=1 / me_range=16 / chroma_me=1 / trellis=1 / 8x8dct=1 / cqm=0 / deadzone=21,11 / fast_pskip=1 / chroma_qp_offset=-2 / threads=22 / lookahead_threads=3 / sliced_threads=0 / nr=0 / decimate=1 / interlaced=0 / bluray_compat=0 / constrained_intra=0 / bframes=3 / b_pyramid=2 / b_adapt=1 / b_bias=0 / direct=1 / weightb=1 / open_gop=0 / weightp=2 / keyint=150 / keyint_min=76 / scenecut=0 / intra_refresh=0 / rc_lookahead=40 / rc=abr / mbtree=1 / bitrate=2400 / ratetol=1.0 / qcomp=0.60 / qpmin=0 / qpmax=69 / qpstep=4 / vbv_maxrate=2640 / vbv_bufsize=2880 / nal_hrd=none / filler=0 / ip_ratio=1.40 / aq=1:1.00

Audio
ID                                       : 257 (0x101)
Menu ID                                  : 1 (0x1)
Format                                   : AAC LC
Format/Info                              : Advanced Audio Codec Low Complexity
Format version                           : Version 4
Muxing mode                              : ADTS
Codec ID                                 : 15-2
Duration                                 : 26 h 4 min
Bit rate mode                            : Variable
Channel(s)                               : 2 channels
Channel layout                           : L R
Sampling rate                            : 48.0 kHz
Frame rate                               : 46.875 FPS (1024 SPF)
Compression mode                         : Lossy
Delay relative to video                  : -48 ms

Let me know, @eric, what else I can provide.

I hope this helps.

eric · February 20, 2023, 7:28pm

mjitkop:

2023/02/20 01:28:46.031526 [DVR] Error running job 1676872790-194 Hot Bench: The Channels DVR Server had a problem. Press play to try again. (Transcoder Reset): Playlist skipped to a higher sequence (356 -> 378): https://service-stitcher-ipv4.clusters.pluto.tv/stitch/hls/channel/6036e6e7ac69c400072afca2/3063648/playlist.m3u8?terminate=false&sid=15f49efd-2f01-4218-a05c-b0d038da9e54&deviceDNT=0&deviceModel=web&deviceVersion=unknown&includeExtendedEvents=false&serverSideAds=true&appName=web&deviceId=c954cb8f-b0e3-11ed-b04c-70b5e8567a24&appVersion=unknown&deviceType=web&deviceMake=Chrome

This is the problem: Playlist skipped to a higher sequence (356 -> 378)

Due to some sort of caching issue on the part of the stream provider, it did not give a sequential stream to us. One time we fetched the playlist ended at 356, the next time the playlist started at 378. This means we lost the chance to download all of those segments between.

There's nothing we can do here beyond what we did (which just leaves a bit missing in the middle).

For the most part, these Interrupted recordings are things in which the thing Channels was downloading from acted up and all we could do is to reconnect and try again (which we did).

eric · February 20, 2023, 7:30pm

Sorry, I didn't realize you were concerned about the duration, please upload the file here and we'll see if there's something that didn't happen right on the re-record:

https://www.dropbox.com/request/XBZmu4iplK3OmoWa3f1K

mjitkop · February 20, 2023, 7:31pm

Thanks, I will start uploading now.

What about that binary file? Any ideas?

eric · February 20, 2023, 7:36pm

You can delete that second file. I see that it was caused because of an issue replacing the original file after rewriting it:

2023/02/20 01:31:18.200671 [ERR] Generating video index for file-18340 failed: replace D:\DVR\TV\Hot Bench\.Hot Bench S01E177 2014-01-01 Episode 177 2023-02-20-0059.mpg.1103564497 D:\DVR\TV\Hot Bench\Hot Bench S01E177 2014-01-01 Episode 177 2023-02-20-0059.mpg: Access is denied.

We'll have to work on improving that cleanup process.

mjitkop · February 20, 2023, 7:37pm

Understood. Thank you.

mjitkop · February 20, 2023, 7:42pm

Upload finished. Thank you for your time and attention to this issue.

eric · February 20, 2023, 8:27pm

Your report was very helpful and identified an issue with how we are writing timestamps on interrupted recordings. Thanks!

eric · February 20, 2023, 8:38pm

The duration issue with interrupted recordings has been fixed:

mjitkop · February 20, 2023, 9:05pm

Nice!

I'm glad I could help!

system · February 24, 2024, 1:36am

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.