ESPN recording issues this weekend (Hulu TVE)

Channels DVR Server TV Everywhere
#1

This is the error message I saw in log for each Formula one recording on Friday, Sat, and Sunday. No observable issues until the Grand Prix Sunday show where stream skipped and has video acceleration playback issues in places. This is recorded by Hulu via TVE.

All recordings were marked as interrupted in the channels client UI, not sure I saw any interruptions on fridays and saturdays recordings, only sundays.

Failing: astits: fetching next packet failed: astits: fetching next packet from buffer failed: astits: reading 188 bytes failed: read /volume1/ChannelsDVR/Streaming/channels-cache-4225873137/313-a20b983315c3145bb393abe65cd803f594443918bff28680d71d5a83773a737a: file already closed

Seeing this as well
Buffer statistics for "TV/Formula 1 Racing/Formula 1 Racing Belgium Grand Prix 2023-07-30-0552.mpg": buf=9%,0%-100% drop=10%

Let me know if more logs are needed.
Logs submitted incase it helps 0104e9b1-403a-441c-98ff-be5ba5820485

Issues with the ESPN channels
#2

First glance, it looks like your drive can't keep up with the incoming stream. This is usually an indicator that your drive might be starting to fail, or that it is under some other intensive operation at the time of the errors.

#3

Thanks, utilization showed 65% during one part of one recording, during others it was below 25%

This is on a synology, shr volume.

Any specific tests you recommend on the disks?

#4

Was that disk activity? Memory? CPU?

Is the DVR running on the NAS, too? Or are your volumes mounted over the network? (If it is local, it sounds like a drive is starting to fail; how old are the drives? IIRC, across drive brands and averaged out, MTBF is roughly 4 years. If you are approaching that age for your drives—or have passed it—now is the time to start thinking about swapping out your drives one at a time to give the RAID time to repair itself.)

#5

Disk activity, that’s what you mentioned.
There is zero evidence any of the 3 drives are failing, no sector errors, etc.
Haven’t done a scrub in a while, will do that.
Yes drives and DVR are on the NAS, locating those in different places is a bit silly IMO.
Drives have 1M hours MTBF based on 220TB/yr written. These drives are nowhere close to MTBF or warranty of 5 years. A single failing drive in the volume would be replaced with the hot spare if significant issues are found. No drive has even minor issues.

I am testing with an OTA recording and a Hulu TVE to see if the issue reproduces on both, is specific to Hulu, or specific to ESPN.

Other posts on this error also indicate issues with the TVE stream formats.

#6

Well, with the minimal information you have provided, I'm afraid I've offered all the help I can.

I assume you have submitted diagnostics from the server and followed up with an email to support, so I'll let the devs continue this thread.

#7

What information were you expecting I did not provide?

#8

So far OTA and other HULU TVE are ok, seems only the ESPN feeds generate the issue. Will do longer soak test and see if I can generate the error on ESPN again (isn’t happening on OTA tuner or BET channel on TVE (random channel that had a movie).

#9

The other option is that this is a network issue (which from your past posts you are familiar with):

The references to astits may make one look to the actual transport stream as the issue. The buffering issues in the logs indicate drive problems, but TS issues may lean more towards the network.

Of course, my armchair diagnosis doesn't have the details that the devs would with your submitted logs.

1 Like
#10

Thanks, my testing seems to indicate this was unique to the ESPN F1 streams, nothing else.

#11

Small clarification, this seems to be a pure ESPN HD issue, New York liberty at Los Angeles WNBA game hit same issue.

#12

Have you tried enabling the Experimental TV Everywhere MPEG-TS Rewriter
in the DVR web admin UI

#13

No. What does it do?
i understand the that is rewrites the stream, but how is it different from nor re-writing the stream?

#14

All TVE (and M3U HLS) streams are rewritten by Channels DVR. The Experimental TV Everywhere MPEG-TS Rewriter is just a newer version being tested that has fixed some problems that appear when using the current default one.

#15

This is indicating that the DVR was unable to write to the filesystem fast enough. There's likely a problem with your disk.

Update: Sorry, that was incorrect. Looking further at your diagnostics, it appears there were many timeouts trying to download video segments from the TVE provider which was what caused the buffer to show it was full.

Can you make sure you aren't running into any local or remote networking issues?

1 Like
#16

I’m also experiencing buffering while streaming ESPN on Apple TV. It buffers every 30 seconds

#17

The only HULU VE channel this seems to be affecting is ESPN HD
I recorded simultaneous movie streams from HULU TVE TNT and TNTP and they are perfect.

I am not aware of any specific local or remote issues. I checked every thing i can think of.
In general almost all my devices are wired, jitter and loss look ok. Low pings. Both internally and externally. Extremely high and consistent internet bandwidth.

I haven't tried reproducing since switching to the rewriter, let me know if you would like me to test again.

These later diags should show the difference between the failing ESP HD and ok TNT/TNTP recordings. Hopefully you can see if i am doing anything wrong in my testing...

6da18128-8204-4c22-bb61-63773059607e .

#18

The latest DVR pre-release has added some additional logging that will make it easier to see when timeouts downloading segments from the TVE CDN are responsible for problems.

Unfortunately when those things happen, it is still impossible for us to know if the issue is local to your network, an issue with the connection between your ISP and the CDN, or a problem between the CDN and the TVE origin encoder.

1 Like