TBS 'Connection Lost' on LAN and Remote

Channels DVR Server TV Everywhere
#1

Server Version: 2024.02.06.0639

It will sometimes play for 2 hours, sometimes 1 hour, sometimes 15 minutes, etc. I just submitted logs from my onn 4K streaming box here on the LAN. i also see this in my Logs from the last time it just did a Connection Lost...

2024/02/07 18:11:01.279512 [SNR] Rewriter statistics for 192.168.1.150 (Office) for ch6033 TBS: discontinuity_detected=0 transport_errors=0 saw_pcr=true saw_pmt=true highest_pts=3421.704667
2024/02/07 18:11:01.280066 [SNR] Buffer statistics for 192.168.1.150 (Office) for ch6033 TBS: buf=0% drop=0%
2024/02/07 18:11:01.280066 [SNR] Streaming statistics for 192.168.1.150 (Office) for ch6033 TBS: timeouts=4 segment_timeouts=4 playlist_timeouts=0
2024/02/07 18:11:01.311138 [TNR] Closed connection to TVE-Hulu for ch6033 TBS

1 Like
#2

From the client logs:

02-07 18:11:01.119 25511 25624 I hdhr_read: error: Timeout after 32s

The client gave up waiting after 32 seconds of not receiving any more data

From the server logs:

2024/02/07 18:11:01.280066 [SNR] Streaming statistics for 192.168.1.150 (Office) for ch6033 TBS: timeouts=4 segment_timeouts=4 playlist_timeouts=0

Notice the segment_timeouts=4 saying that there were 4 timeouts while trying to download a video segment.

and elsewhere in the more detailed server logs:

at=2024-02-07T18:11:01.2795129-06:00 tag=tbs action=fetch url="https://tve-live-aka.warnermediacdn.com/hls/live/2023173/tbseast/noslate/1706731563_video_720p-30fps-5000kbps/video_103936.ts?..." sequence=1735936 continuity_counter=86 initial_status=segmentDownloading program_date_time=2024-02-08T00:10:01.656Z duration=6.006 init_time=11.0063ms request_time=16.061ms content_length=3902896 timedout=bodyTimer downloaded_length=2420688 failed=temporary failed_while=reading err="hls: timeout while waiting for data" first_byte_time=18.9947ms last_byte_time=11.5830505s max_read_time=1.3224637s body_time=12.01265s time=12.0316447s status=segmentTemporarilyFailed

We can see the last attempt to download one of the segments gave up after 12s.

at=2024-02-07T18:11:01.2795129-06:00 tag=tbs action=read sequence=1735936 offset=0 buf.cap=32768 n=0 err="failed after 40.2817804s: hls: temporary failure while downloading: hls: timeout while waiting for data" time=40.2817804s

And the entire HLS session gave up after 40 seconds of not succeeding in downloading a new segment.

You're either running into resource issues on your server/network or issues between your ISP and the streaming servers. Running the Resource Intensive Troubleshooting will give you an idea if you have local issues.

You can setup something like PingPlotter against tve-live-aka.warnermediacdn.com and see if you see any failures between your ISP and the CDN.

#3

@eric thanks for looking. I just ran the resource intensive troubleshooting and then submitted logs:

d0ba25af-33c9-4d29-836c-12f29d959df1

running ping tests now and will report back any findings thanks.

#4

It just dropped again. Ran continuous ping to tve-live-aka.warnermediacdn.com

Ping statistics for 23.195.81.24:
Packets: Sent = 2024, Received = 2024, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 2ms, Maximum = 92ms, Average = 3ms

From the Logs:

2024/02/07 20:08:46.072571 [SNR] Rewriter statistics for 192.168.1.150 (Office) for ch6033 TBS: discontinuity_detected=0 transport_errors=0 saw_pcr=true saw_pmt=true highest_pts=4208.554033
2024/02/07 20:08:46.073116 [SNR] Buffer statistics for 192.168.1.150 (Office) for ch6033 TBS: buf=0% drop=0%
2024/02/07 20:08:46.073116 [SNR] Streaming statistics for 192.168.1.150 (Office) for ch6033 TBS: timeouts=7 segment_timeouts=7 playlist_timeouts=0
2024/02/07 20:08:46.115527 [TNR] Closed connection to TVE-Hulu for ch6033 TBS
2024/02/07 20:08:46.115527 [TNR] Error during live stream for ch6033 TBS: failed after 38.9626917s: hls: temporary failure while downloading: hls: timeout while waiting for data

#5

It could just be that the CDN servers themselves are overwhelmed and are sending data slowly. There's really nothing that we can do about these timeouts, and it appears there's nothing local to your network that is likely the cause, so this is just the state of streaming on the internet.

#6

Had the same issue a few days ago with two recordings while a third recording happening at the same time wasn't affected.

2024/02/03 04:13:07.113874 [DVR] Error running job 1706961600-74: failed after 36.825867161s: hls: temporary failure while downloading: hls: timeout while waiting for data
2024/02/03 04:13:25.079201 [DVR] Error running job 1706961600-72: failed after 32.237773997s: hls: temporary failure while downloading: hls: timeout while waiting for data
#7

Hmmmm ok. Just happened again with no ping loss. Oh well...

#8

This :point_up:

I imagine we'll see more of this on Sunday with all the Super Bowl streaming going on.

#9

"And the entire HLS session gave up after 40 seconds of not succeeding in downloading a new segment."

Is there a way to change it to 60 or 90 seconds before it gives up and throws an error? Or better yet maybe force a retry or an automatic reconnect? Something like "Connection Timeout".... 'Attempting to Reconnect' and it auto-reconnects? Or at least a setting to enable/disable such action?

Just a thought. Thanks for all the great work.

#10

Also use pingplotter instead of ping.exe in windows. Pingplotter will give you statisitics for each hop. You may have packet loss to one of the hops with your isp or something .

2 Likes
#11

No, if it's taking that long, there's something wrong enough that it warrants giving up.

There are a number of reasons why this isn't something we're going to do, but to just explain one of them: When playback restarts, the old buffer is lost. If we automatically reconnected when you had gotten up for a moment, it could happen that you would come back and have no idea why you can't rewind anymore.

When you see the error, does it instruct you to press Play to restart? Does it work when you do that?

#12

1000006262
10000062621870×1424 351 KB

I hit play and it starts right back up.

#13

https://winmtr.net/

This generates output that is more forum friendly for copy/paste (and it's free)

WinMTR Statistics

WinMTR statistics

Host % Sent Recv Best Avrg Wrst Last
OPNsense.lan 0 27 27 0 0 2 0
REDACT 0 27 27 8 10 25 10
REDACT 0 27 27 10 13 28 21
REDACT 0 27 27 9 14 25 17
lag-27.rcr01lsancarc.netops.charter.com 0 27 27 11 13 29 14
lag-26.lsancarc0yw-bcr00.netops.charter.com 0 27 27 12 14 25 15
066-109-009-199.inf.spectrum.com 0 27 27 11 13 18 13
ae9.ctl-lax2.netarch.akamai.com 0 27 27 12 24 75 52
a184-50-27-50.deploy.static.akamaitechnologies.com 0 27 27 12 14 25 14
1 Like
#14

I posted in your other TBS thread yesterday that my TBSHD (TVE-Hulu) is down. It's day 2 and it still doesn't work. You replied with some eyes :eyes:. I am not sure what that means?

Link here: No TBS West. Lost local Fox and discovery channels. YTTV - #10 by troyhough

This is the error I am getting today still:

2024/02/09 10:51:03.740293 [ERR] Could not start stream for TVE-Hulu ch6033 TBS: TVE: Could not fetch playlist from turnerlive.akamaized.net: GET: https://turnerlive.akamaized.net/hls/live/572096/tbse/noslate/tv/master.m3u8?hdnts=exp=1707497762~acl=/hls/live/*~id=b716f441706eaaf7a6e330d18df8f918-1707497763~hmac=0eb380dcb3ea20420ac242728766d4e189ee7d01c3c333848a57ff87687289cb: 403 Forbidden

#15

We are looking into it but it appears these channels may not be available to Channels DVR any more.

#16

Thanks for replying @tmm1 I have since updated to 2024.02.09.0659 and it worked immediately after. Not sure if it was corrupt/stuck but all good now!

#17

Ok TBSHD just did another "Connection Lost". I'll post the log as well as the WinMTR stats for the entire duration.

2024/02/09 12:54:41.008787 [SNR] Rewriter statistics for 192.168.1.150 (Office) for ch6033 TBS: discontinuity_detected=0 transport_errors=0 saw_pcr=true saw_pmt=true highest_pts=3452.862233
2024/02/09 12:54:41.008787 [SNR] Buffer statistics for 192.168.1.150 (Office) for ch6033 TBS: buf=0% drop=0%
2024/02/09 12:54:41.008787 [SNR] Streaming statistics for 192.168.1.150 (Office) for ch6033 TBS: timeouts=2 segment_timeouts=2 playlist_timeouts=0
2024/02/09 12:54:41.047390 [TNR] Closed connection to TVE-Hulu for ch6033 TBS
2024/02/09 12:54:41.047390 [TNR] Error during live stream for ch6033 TBS: failed after 33.9231296s: hls: temporary failure while downloading: hls: timeout while waiting for data

|------------------------------------------------------------------------------------------|
| WinMTR statistics |

Host - % Sent Recv Best Avrg Wrst Last
192.168.1.1 - 0 5403 5403 0 0 9 0
REDACT - 0 5401 5401 0 3 75 1
REDACT - 0 5402 5402 0 3 112 2
REDACT - 0 5401 5401 2 4 94 4
206.126.235.4 - 0 5401 5401 3 24 645 8
a23-195-81-48.deploy.static.akamaitechnologies.com - 0 5401 5401 2 3 168 3
________________________________________________ ______ ______ ______ ______ ______ ______

WinMTR v0.92 GPL V2 by Appnor MSP - Fully Managed Hosting & Cloud Provider

#18

Are you watching the channel while recording it?
2024/02/09 12:54:41.008787 [SNR] Rewriter statistics for 192.168.1.150 (Office) for ch6033 TBS: discontinuity_detected=0 transport_errors=0 saw_pcr=true saw_pmt=true highest_pts=3452.862233
highest_pts=3452.862233 = 57 minutes, 32 seconds

2024/02/09 12:54:41.047390 [TNR] Error during live stream for ch6033 TBS: failed after 33.9231296s
33.9231296s = 33.9 seconds

Appears their CDN edge server may be having issues. Pinging w/ICMP and streaming HTTPS TCP/IP are different.

#19

I am not recording it. Just watching it. Is there a diff address to ping test?

#20

This is not a ping level issue. They are changing their TVE streams and these old ones are not working reliably anymore. The new ones are DRM.

1 Like
Tru TV TVE Error