Recording Interrupted Error

JWIII · February 3, 2023, 6:59pm

I've received the "recording was interrupted" error on some recordings the last few days. In the DVR server, some recordings have the message: "Failed: astits: fetching next packet failed: astits: fetching next packet from buffer failed: astits: reading 188 bytes failed: hls: segment removed." What does this mean?

I'm running the server on a M1 MM and use Channels primarily on Apple TV. I recently switched hard drives and am using Fubo and DTV Stream (backup) as TVE providers.

What can I do to fix these errors?

tmm1 · February 3, 2023, 7:02pm

~~"segment removed" means a file was deleted out of the Streaming directory unexpectedly.~~ Maybe something is up with your new drive.

EDIT: @eric may have a better idea, and we'd also need diagnostics to be able to see the context of what caused the error

eric · February 3, 2023, 7:33pm

Clarification: The segment removed means that the segment that we were trying to read was removed from the TVE provider's m3u8 we are reading from (the live m3u8 contains a rolling buffer of segments, new ones get added, old ones get pruned). This generally only happens when the drive we are recording to is so slow that we get minutes behind in recording. My best guess is the drive you are writing to is being overwhelmed, but as @tmm1 said, we would need diagnostics to know for sure.

JWIII · February 3, 2023, 8:46pm

Thanks @eric and @tmm1! So... looking at my setup, I realize I screwed up. I added my new drive as a storage path instead of replacing it as the DVR, so recordings were still being made to the old (slower) drive. I've switched it around now so that new recordings will be made to the new drive and any previous recordings will play from the old drive (there's no issue with doing it this way... changing the DVR storage without migrating the full folder from the older drive, correct?).

I've set up a bunch of random recordings for tonight. If I see any error messages, I'll post and submit diagnostics. Thanks!

channelsjimt · April 24, 2023, 8:52pm

I'm in my first month as a Channels DVR subscriber, and am very pleased with Channels' look, feel, functionality, and performance. After recording about 20 movies sourced from TCM on TVE, I've seen two of those movies flagged with the "Recording was interrupted" warning. In each case, I didn't see an actual interruption in the recording. In both of those cases, though (and in no other instance), I saw the following "Could not fetch playlist" message in the logs:

2023/04/20 20:14:30.000588 [DVR] Starting job 1682043270-ch6039 Mean Streets (1973) on ch=[6039]
2023/04/20 20:14:36.709525 [TVE] stream timestamps: tcm: start_at=2023-04-20T20:04:30-06:00 end_at=2023-04-20T20:14:18-06:00 live_delay=11.94151982s
2023/04/20 20:14:36.709679 [TNR] Opened connection to TVE-Comcast_SSO for ch6039 TCM
2023/04/20 20:14:36.709840 [DVR] Recording for job 1682043270-ch6039 from TVE-Comcast_SSO ch6039 into "Movies/Mean Streets (1973) 2023-04-20-2014.mpg" for 2h0m29.9992737s
2023/04/20 20:14:39.988532 [IDX] Generating video index for job 1682043270-ch6039
2023/04/20 21:45:03.724671 [SNR] Buffer statistics for "Movies/Mean Streets (1973) 2023-04-20-2014.mpg": buf=0% drop=0%
2023/04/20 21:45:04.395807 [MTS] Statistics for "Movies/Mean Streets (1973) 2023-04-20-2014.mpg": skipped=0 unhandled_packets=0 discontinuity_detected=0 transport_errors=0 invalid_pts=0 invalid_dts=0 saw_pcr=true saw_pmt=true highest_pts=5363.888233
2023/04/20 21:45:04.480297 [TNR] Closed connection to TVE-Comcast_SSO for ch6039 TCM
**2023/04/20 21:45:04.480336 [DVR] Error running job 1682043270-ch6039 Mean Streets (1973): Could not fetch playlist: https://turnerlive.akamaized.net/hls/live/2023186/tcmeast/noslate/VIDEO_1_5128000.m3u8?hdntl=exp=1682072076~acl=%2fhls%2flive%2f*~hmac=0a05ddcef0026612842c7bf8a8390fd27b822e79423e91127199a670b103a91c: Get** "https://turnerlive.akamaized.net/hls/live/2023186/tcmeast/noslate/VIDEO_1_5128000.m3u8?hdntl=exp=1682072076~acl=%2fhls%2flive%2f*~hmac=0a05ddcef0026612842c7bf8a8390fd27b822e79423e91127199a670b103a91c": http2: timeout awaiting response headers**
2023/04/20 21:45:06.673026 [DVR] Starting job 1682043270-ch6039 Mean Streets (1973) on ch=[6039]
2023/04/20 21:45:11.076573 [TVE] stream timestamps: tcm: start_at=2023-04-20T21:35:06-06:00 end_at=2023-04-20T21:44:54-06:00 live_delay=10.444568179s
2023/04/20 21:45:11.076747 [TNR] Opened connection to TVE-Comcast_SSO for ch6039 TCM
2023/04/20 21:45:11.076832 [DVR] Recording for job 1682043270-ch6039 from TVE-Comcast_SSO ch6039 into "Movies/Mean Streets (1973) 2023-04-20-2014.mpg" for 29m53.326808646s
2023/04/20 22:15:10.517589 [TNR] Closed connection to TVE-Comcast_SSO for ch6039 TCM
2023/04/20 22:15:11.050779 [SNR] Buffer statistics for "Movies/Mean Streets (1973) 2023-04-20-2014.mpg": buf=0% drop=0%
2023/04/20 22:15:11.957821 [MTS] Statistics for "Movies/Mean Streets (1973) 2023-04-20-2014.mpg": skipped=0 unhandled_packets=0 discontinuity_detected=0 transport_errors=0 invalid_pts=0 invalid_dts=0 saw_pcr=true saw_pmt=true highest_pts=7251.204322
2023/04/20 22:15:11.958028 [DVR] Finished job 1682043270-ch6039 Mean Streets (1973)
2023/04/20 22:15:13.549590 [DVR] Processing file-71: Movies/Mean Streets (1973) 2023-04-20-2014.mpg
2023/04/20 22:15:16.767083 [IDX] Generating video index for file-71: Movies/Mean Streets (1973) 2023-04-20-2014.mpg
2023/04/20 22:15:18.868007 [DVR] Running commercial detection on file 71 (Movies/Mean Streets (1973) 2023-04-20-2014.mpg)
2023/04/20 22:15:52.395007 [IDX] Finished video index generation for file-71 in 35s
2023/04/20 22:41:24.396612 [DVR] Commercial detection for Mean Streets (1973) 2023-04-20-2014.mpg finished with 6 markers in 26m5.528863398s.

Both recordings appear normally in playlists, along with their respective expected art, and (again) with no interruption in either recording. Just what is Channels' reliance on the "playlist" from a provider? Is it not uncommon for the playlist fetch to fail, but yet there be no evident indication of that (other than Channels' "Recording was interrupted" flag)?

Thanks.

tmm1 · April 24, 2023, 8:55pm

This is not normal and indicates some type of internet issue between you and the streaming video CDN. Are you using any proxies, custom DNS or VPN?

The DVR will attempt to reconnect and there may be a slight glitch or audio/video replay at the moment of the interruption.

channelsjimt · April 24, 2023, 9:01pm

Thanks for the (super) quick response! Nope, I'm not using a VPN or a custom DNS (I'm relying on my ISP's DNS servers).

The stats for both connections appear to be good, although I don't know what the highest_pts stat is telling me:

2023/04/20 21:45:04.395807 [MTS] Statistics for "Movies/Mean Streets (1973) 2023-04-20-2014.mpg": skipped=0 unhandled_packets=0 discontinuity_detected=0 transport_errors=0 invalid_pts=0 invalid_dts=0 saw_pcr=true saw_pmt=true highest_pts=5363.888233

2023/04/16 23:43:38.080000 [MTS] Statistics for "Movies/The Wild Bunch (1969) 2023-04-16-2244.mpg": skipped=0 unhandled_packets=0 discontinuity_detected=0 transport_errors=0 invalid_pts=0 invalid_dts=0 saw_pcr=true saw_pmt=true highest_pts=3501.121500

chDVRuser · April 24, 2023, 9:35pm

pts = Presentation Time Stamp
When you play a video, it's the playhead (current) position time (where you are in the video).
The numbers you see in the log entries are in seconds and mean how much of the program was recorded when it was interrupted.

highest_pts=5363.888233 means that the recording was interrupted at play time 01:29:23.888233
highest_pts=3501.121500 means that the recording was interrupted at play time 00:58:21.121500

If you position the playback of the recording a few seconds before that interruption time and start playing, you'll either see it jump ahead or repeat when it reaches that time position.

sdust · April 24, 2023, 9:59pm

Could we increase the buffer size? I have 8GB of memory waiting to be used for something.

tmm1 · April 24, 2023, 10:08pm

It could be ipv6 vs ipv4 issue. You could setup something like PingPlotter against this CDN (turnerlive.akamaized.net) to measure latency over time.

channelsjimt · April 25, 2023, 12:51am

Helpful--thx, @chDVRuser. For the first recording, there was a skip of about 10 seconds at the pts, although the skip's impact on the resulting video & audio was virtually imperceptible (... at least, without having a script to follow along with the dialog). The impact at the second recording's pts was more noticeable, with my attention directed to that point in the movie.

channelsjimt · April 25, 2023, 1:08am

thx, @tmm1. I've downloaded PingPlotter and am tracing my 12 hops or so to turnerlive.akamaized.net. All hops look good but one--hop 5 to hop 6, where both endpoints, about 60 miles apart, are owned by my ISP. That hop is showing an alarmingly high packet loss percentage. I'm hoping that's an interim thing. If not, I'll be in contact with them.

eric · April 25, 2023, 3:48am

What is it that you think needs to be accommodated? The segments that are referenced in the above post aren't stored in RAM, they're stored on disk.

When we fetch an m3u8 playlist, it has a list of segments. Normally that playlist will contain minutes worth of segments. We try to download those segments as fast as the system allows. If the segments are downloading so slowly that the segment that is multiple minutes behind realtime cannot be downloaded before it no longer exists in the fetched m3u8 playlist, there is little chance that anything will help it recover, so we fail.

system · April 24, 2024, 3:49am

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.