Stream error during commercial break TVE

I'm running 2019.09.03.1947. I've had 2 error types. The 403 as previously mentioned and a 404 on the MLB Channel.

2019/09/03 18:28:10 [TNR] Opened connection to TVE-DTV for ch6193 MLBTV [transcode=none]
2019/09/03 18:34:53 [ERR] Error during stream TVE-DTV ch6193 MLBTV: Failed to download file: http://mlblive-akc.mlb.com/ls01/mlbam/mlb_network/NETWORK_LINEAR_1/MLB_VIDEO_MLBN_FAUX_LINEAR_STREAM_1_MLBN_Linear_Stream_Tuesday_20190903_1567490547777/3000K/01/34/09.ts: 404 Not Found
2019/09/03 18:34:53 [TNR] Closed connection to TVE-DTV for ch6193 MLBTV
2019/09/03 18:35:21 [TNR] Opened connection to TVE-DTV for ch6073 FNC [transcode=none]
2019/09/03 18:40:22 [ERR] Error during stream TVE-DTV ch6073 FNC: Failed to download file: https://x-live-fox-stgec.uplynk.com/ausc/slices/afd/4f59ea65df184a2580ad8fc80d3eaa32/afd76bf662454d8b870fd07499a57c8f/H0000000F.ts?pbs=0b54c4c92fbe4b0ba60b9579ad77daf9&_jt=l&chid=1e298c9d65a245b3bd37f339cadcca58&oid=5b6fd7c6528d417e8bdad7f79754bb9e&si=8: 403 Forbidden
2019/09/03 18:40:22 [TNR] Closed connection to TVE-DTV for ch6073 FNC
2019/09/03 18:44:07 action=p12_validate status=200 OK

It did fail a second time for me last night. In this case it failed as “normal” meaning the screen did not go black on me.

Logs:

2019/09/03 21:39:35 [TNR] Opened connection to TVE-sony_auth-gateway_net for ch6073 FNC
2019/09/03 21:59:04 [ERR] Error during stream TVE-sony_auth-gateway_net ch6073 FNC: Failed to download file: https://x-live-fox-stgec.uplynk.com/ausc/slices/afd/4f59ea65df184a2580ad8fc80d3eaa32/afd76bf662454d8b870fd07499a57c8f/H0000000F.ts?pbs=fa0839636d2442b99070e28b2022269f&_jt=l&chid=1e298c9d65a245b3bd37f339cadcca58&oid=5b6fd7c6528d417e8bdad7f79754bb9e&si=25: 403 Forbidden
2019/09/03 21:59:04 [TNR] Closed connection to TVE-sony_auth-gateway_net for ch6073 FNC
2019/09/03 21:59:19 [TNR] Opened connection to TVE-sony_auth-gateway_net for ch6073 FNC

Downloaded and installed latest this morning. Ran for about 90 min ok and then it encountered the error again.

Log:

2019/09/04 06:53:13 [SYS] Downloading new version v2019.09.04.0522
2019/09/04 06:53:16 [SYS] Update downloaded and verified successfully.
2019/09/04 06:53:18 [TNR] Closed connection to 10711025 for ch11.1 WTVD-D1
2019/09/04 06:53:42 [SYS] Shutting down...
2019/09/04 06:53:42 [DVR] Recording engine stopped.
2019/09/04 06:53:42 [SYS] Bonjour service stopped.
2019/09/04 06:53:42 [SYS] Goodbye.
2019/09/04 06:53:43 [SYS] Starting Channels DVR v2019.09.04.0522 (darwin-x86_64 pid:30902) in /Users/admin/Library/Application Support/ChannelsDVR/data
2019/09/04 06:55:05 [HDR] Found 1 devices
2019/09/04 06:55:07 [SYS] Started HTTP Server
2019/09/04 06:55:07 [DVR] Recording engine started in /Volumes/Channels-DVR
2019/09/04 06:55:07 [DVR] Waiting 6h34m52.733589s until next job 1567618200-35 The Bold and the Beautiful
2019/09/04 06:55:07 [SYS] Removing old update 2019.08.30.1336
2019/09/04 06:55:07 [NAT] Successfully mapped port 8089 using natpmp
2019/09/04 06:55:07 [SYS] Bonjour service registered with XXXXX.XXXXXXXXXX.XXX
2019/09/04 06:55:17 [IDX] Pruned 198 expired airings from USA-OTA27587 in 15.632726ms.
2019/09/04 06:55:17 [IDX] Pruned 807 expired airings from X-TVE in 25.668514ms.
2019/09/04 06:55:42 [SYS] Created database snapshot: backup-20190904.065507
2019/09/04 06:55:42 [SYS] Removing old backup backup-20190816.144516
2019/09/04 06:55:47 [TNR] Opened connection to 10711025 for ch11.1 WTVD-D1
2019/09/04 07:16:30 [TNR] Closed connection to 10711025 for ch11.1 WTVD-D1
2019/09/04 07:16:42 [TNR] Opened connection to TVE-sony_auth-gateway_net for ch6073 FNC
2019/09/04 07:55:07 [NAT] Successfully mapped port 8089 using natpmp
2019/09/04 08:23:01 [TNR] Sharing existing connection to TVE-sony_auth-gateway_net for ch6073 FNC (clients=2, len=0)
2019/09/04 08:48:19 [ERR] Error during stream TVE-sony_auth-gateway_net ch6073 FNC: Failed to download file: https://x-live-fox-stgec.uplynk.com/ausc/slices/afd/4f59ea65df184a2580ad8fc80d3eaa32/afd76bf662454d8b870fd07499a57c8f/H0000000F.ts?pbs=cdeea231d5dd4c4195ebcbe25c40dbb9&_jt=l&chid=1e298c9d65a245b3bd37f339cadcca58&oid=5b6fd7c6528d417e8bdad7f79754bb9e&si=67: 403 Forbidden
2019/09/04 08:48:19 [TNR] Closed connection to TVE-sony_auth-gateway_net for ch6073 FNC

I see in my logs we've had 8 occurrences of this error in the last 2 days. Google searching "uplynk 403 forbidden" yielded some interesting posts about this error related to uplynk and ffmpeg. Technically, outside of my scope of understanding but maybe it will help in tracking down the issue.

@tmm1, I just came home from a week long trip to California and one of the first things my wife said to me is that it’s time to find an alternative because she’s tired of this issue.

Honestly, I’m not there at this point but I have to say I feel like for the most part this problem is being ignored as if it’s not a big deal.

To us it is a big deal. It’s disruptive and frustrating to watch the focus on new features and troubleshooting of other’s network / setup issues rather than fixing the issues that exist for the users.

Channels is / has always been a great platform. A significant part of that greatness is the support. After a month of this problem I have to be blatantly honest and say I’m not “feeling the love” right now.

This is happening a lot on Fox News and Fox Business for me in the last week or so. I normally download nightly builds so Im not sure if that is the issue or not.

Those are the 2 stations that seems to be the problem children. I don't know if anyone has looked to fix this yet

That stuff happen in the apps. Its buggy as snot and those same annoying commercials are even worse.

We have to do all of those things. You're all our customers and we can't neglect users with installation issues, new features that make the software easier to use and more stable, or strange and flaky issues with certain TVE channels.

In the case of TVE channels, there are a lot of things we can do to make it better and more stable, but at the end of the day, we're at the mercy of the TV networks providing a good and stable video stream.

We've put a lot of work into this issue (as we do all bugs that are reported), but in this case haven't found any simple answers for why this happens. It isn't clear to us if there's an issue with our integration with the channels or if there's just a fundamental issue with the hosting. If these sorts of glitches happen when watching on the Fox website or the Fox app, it's likely a bug in how they're serving their video. If that's the case, there's a limit to how much we can do to fix things.

1 Like

@eric, I hear you, but frankly disagree on some of your points.

First, I’ve not seen it fail in the browser version. In fact, I watched FNC for several hours yesterday on an airplane of all places and it ran just fine. Better than in channels from the perspective of this particular error.

Second, you do not have to support users resolving their own network, external disk drive, Nas, etc errors and issues. It’s very nice that you do, but the reality is you need to support your product to the extent that you show the issue is NOT your product. The community is the proper place to support those other issues.

Thirdly, you’ve spent far more time IMHO working on new features (transcoders) than this issue. Maybe I’m wrong but I can only give you my impression based on the activity in the forums.

Lastly, did you do the search that I suggested earlier in the week? There are numerous reports of this error against this server in the ffmpeg forums (you use ffmpeg, right?) with solutions to them.

You’re right that you’re at the mercy of what the stream providers send, but only to a point. You’re responsible for how your program reacts to and handles errors.

I say all of this from the perspective of someone who has been responsible for providing technical support for 4+ fortune 100 tech companies over the last 25 years.

Here are a few suggestions for you guys. You don’t have to follow any of them, but I’m going to put them out there anyway because I love your product, want to see it succeed even more, and will continue to recommend it.

  1. Listen to your user community and respond to them in some way but do what you can to do it more efficiently. There are a lot of requests for features that seem to fall on deaf ears (ie; changing the colors, clocks, PIPs, overlays of guide data). Put together an FAQ that establishes your position on these types of things and point users to them. You used to do this for DRM as an example.

  2. Put together a solid self-help guide. This can be jointly maintained by you and your users. New users can leverage this as opposed to writing you directly. And, if you have this then you can quickly point users to this.

  3. I’ll reiterate, questions on recommended hardware, network issues, etc. are not in your realm of direct responsibility. Your users and an FAQ can handle this.

  4. Put some controls in place around the sprawl. Channels started out, I believe, as primarily an Apple ecosystem focused solution. Over the years its grown tremendously in what it supports. This grows the user base but it means you’re spread even more thinly. My point is, maybe address some of the current issues/requests before moving to support ARM devices or RPi4s.

All of these are meant as suggestions as to how you can streamline your time so that you can focus more on your product. One thing we all know is that when it comes to TV it’s a very different world in end user computing. The sensitivity to issues or tolerance of them is almost akin to stock trading. It’s just not the same as your typical app. You need to continue to improve over time in order to stay ahead of your competition. Providers such as Hulu, PS Vue, Sling, etc will eat your lunch when it comes to ease of use. You’re better than them now, but don’t rest on that. If you allow them to they will outpace you from a feature perspective over time and you will lose share to them. Right now, you’re ahead because of your ability to store recordings, skip commercials and your support, but you’re behind on things like ease of use, complexity and the limitation to add features do to your custom video player. You’re a small company competing against giants you’re continued success depends on your ability to innovate a solid and easy to implement product. You’re there now for the most part but I sense you’re at a critical juncture and need to adapt to the size that you’ve become.

We (my family) love Channels and want nothing but it’s continued success.

Good thing you're not a Tablo customer. :flushed:

TVE is still in BETA and is awesome. I'm sure they're on it.

I WAS a Tablo customer. Emphasis on the most important word. Key thing here is I want to remain a Channels customer, but if the WAF can't be addressed then something will need to change (hoping not).

That makes two of us regarding Tablo. Didn't it take over 3 years to implement Dolby 5.1? Six months for buggy firmware?

I figured it was the WAF. My married friends tell me to stay single...one just this week as a matter of fact.

I have A LOT of issues with the standalone Fox News app before TVE. One thing your wife may want to try in the meantime is the FOXNOW app. It seems to have less issues for me compared to the standalone app.

DVRing works great and don't seem to have issues that way. Maybe that's something for the Channels team to look into.

That's a great datapoint. The more we learn about the instances that it does or doesn't have problems when people stream the channels from other sources the more we can understand how much we will be able to improve.

Helping people use our product has always been a part of what we do. Understanding how the network and computers and storage interact requires a lot of different skills and many times our customers run into situations that are beyond their knowledge. We want to do what we can to get them over those hurtles and watching TV and so we're going to do what we can to help them.

It's definitely true that I've been focused on the new transcoder recently. That feature in particular is funny to think about because you could think of it as a new feature or you could think of it as a lot of bug fixes. The reason why the new transcoder is happening is that there were a number of frustrating issues that users watching recordings remotely would run into and the new transcoder fixes them.

Another aspect of this is that each of us working on Channels has a different skillset and aspects of the product that we are able to improve the most, so you're going to see work done in multiple aspects of the product happening at the same time. It doesn't mean that nothing is being done on the thing you care about most.

We don't use ffmpeg for our TVE streaming — it's all code we've built ourselves. Any error messages you see from other tools that are similar are likely coincidental.

Yep, that's true. And it's definitely one of the things we've been discussing as to how to solve issues like this. If it were easy to handle we certainly would have, but it would currently require us to change a bunch of stuff to make that happen, so we haven't been able to do that yet.

We appreciate the effort that you put into this response and your thoughts on the market.

We've built something pretty great with a very small team (up to 3 developers with me on board) and we'll continue to improve and make it better. Having a good sense of what to build is as important to a company as having the skills to build it. We wouldn't have gotten this far if we didn't have both.

3 Likes

We've put out atleast half a dozen updates trying to resolve this issue. If there was an easy fix available, it would already have been integrated.

Of the 200+ channels we have added into the TVE beta, only these two seem to have this issue. We have several other channels from fox.com/live that work fine. Dozens others also use the same uplynk backend servers and also have no issues. And even FNC/FBN worked fine for weeks when we first added them, then something changed on their end and these issues started.

Anyway, I know this is really frustrating but we are doing our best to try to figure out what's causing the problem. There are some more possible fixes in v2019.09.14.2143 if you'd like to try it out.

@tmm1

I’m not trying to say you haven’t tried to fix this. To be honest though I’m not aware of a half dozen attempts to fix; maybe that’s a part of my frustration with this. I update every day or so so I’ve got any fixes you tried. Let us know when a new attempt is in beta.

That said, I’ve been running 2143 for the past hour or so and so far so good. Longest I‘ve seen in the past has been about two hours without the error. One thing interesting I’ve seen tonight is we’re seeing more “real” commercials along with the PSA commercials.

Please don’t take my frustration as criticism. At least not as unconstructive criticism. We love your app and really enjoy using it and want it to succeed. If your making changes to help this problem then it would help if you let us know so that we understand that it is being worked on.

I remain 100% committed to this app and will always provide feedback and be willing to test any fix.

Just wanted to add my 2 cents to the discussion, for what it's worth with inflation and all.

I don't watch or record either of these two "problem" TVE channels, although I did record a couple previously to help troubleshoot here

From what I've seen this only happens going into a commercial break or a commercial segment, correct?

If that's true they may be using targeted advertising and switching to another server/stream for the commercials.

Maybe based on demographics of the user (provider used to login to TVE/browser agent headers/zipcode/ip address, etc). And don't forget how Google gets their revenue.

Would it not be helpful if someone experiencing the issue used tuner sharing and while watching the live feed, they also recorded it?

That way you get log info from both the live viewing and the recording.

If it would help, I could do that, but I won't be watching the live stream, just running it in the background with the volume muted.

As long as the DVR web UI doing the live viewing logs the disconnect, I think it would be more data points.

Are there specific shows/times of day/etc. that are more likely to see the issue?

FWIW, and fingers crossed, we’ve been watching for almost 3 hours and haven’t seen the issue. Probably premature to say it’s fixed but it does look good as of now.

:crossed_fingers: If you still have issues with the new version I can setup a test tomorrow.
Just let me know what shows/times to stream live and record. My TVE NAS is not busy tonight or tomorrow, so would be willing to help out by testing.

Started early with v2019.09.15.0041 at 20:47:36 as I didn't see any responses.

Watching (in the background) two channels 6073(FNC) and 6074(FBN) via Channels DVR web UI while recording both those channels for 5+ hours (300 minute end-padding).

Apparently Lou Dobbs Tonight on ch6074(FBN) loses the battle to Watters' World on ch6073(FNC) tonight.

Not anything in the log after;

2019/09/14 21:52:33 [ERR] Error during stream TVE-Comcast_SSO ch6074 FBN: Failed to download file: https://foxvdms.akamaized.net/auso/slices/d73/4f59ea65df184a2580ad8fc80d3eaa32/d7319c379cec434198f1e1fbfcc9f33c/H00000004.ts?pbs=6452022af2604ced8eb5a818acaebd07&_jt=l&chid=0889265100704d178922db41fa6b6b91&oid=5b6fd7c6528d417e8bdad7f79754bb9e&si=65: 503 Service Unavailable
    Last message repeated 1 times

Other than it repeating trying to download again to the end with this message repeated...

2019/09/14 21:52:33 [MTS] Statistics for "TV/Lou Dobbs Tonight/Lou Dobbs Tonight 2003-06-16 2019-09-14-2051.mpg": skipped=8 unhandled_packets=0 discontinuity_detected=69 transport_errors=0 invalid_pts=0 invalid_dts=0 saw_pcr=false saw_pmt=true highest_pts=3750.198778
2019/09/14 21:52:33 [DVR] Job 1568516400-ch6074 Lou Dobbs Tonight ended prematurely: 4h7m26.037939258s
2019/09/14 21:52:34 [DVR] Starting job 1568516400-ch6074 Lou Dobbs Tonight on ch=[6074]
2019/09/14 21:52:34 [TNR] Sharing existing connection to TVE-Comcast_SSO for ch6074 FBN (clients=2, len=0)
2019/09/14 21:52:34 [DVR] Recording for job 1568516400-ch6074 from TVE-Comcast_SSO ch6074 into "TV/Lou Dobbs Tonight/Lou Dobbs Tonight 2003-06-16 2019-09-14-2051.mpg" for 4h7m25.926415771s

6073(FNC) is still displaying live tv, where 6074(FBN) is still frozen in the display showing this

Let me know if you want any logs before I delete the test recordings tomorrow night.
Screenshot_2019-09-14%20Channels%20DVR(1)