Stuttering playback on Apple TV (Windows Server 2019)

@tmm1 Since we have multiple users reporting the issue, is there any testing in your lab that you can do using Windows Server 2019 and an Apple TV? Microsoft offers Windows Server 2019 available for testing purposes.

On the client side we're not seeing much useful stuff in the logs beyond the cache exhaustion discussed earlier. We're trying to figure out what other diagnostics we can add which might help shed light into what's going on inside tvOS.

On the DVR side we have a lot of good diagnostics, but I need another one which is sent from the DVR while the Apple TV video player is still active and stalling during recording playback.

How often is this issue happening? In your last logs I saw you watched a full hour+ recording without problems, then starting the next recording it happened almost right away (buffered 95s and stopped receiving data, then 95s later the video player stalls because the buffer is empty).

Is it happening on recordings only or live tv too?

Recordings only. The issue is very random.

I will submit DVR logs next time the issue happens.

Logs from the DVR uploaded.

Problem is happening as a I type this. Speed test results from DVR to the Apple TV client.

Speedtest

Download
968.41 Mbit/s

Latency
3.05 ms

Jitter
0.23 ms

1 Like

Thanks. I was able to see the DVR is indeed streaming the file out, and shows its waiting on the OS to inform it when the network connection is ready to accept more data.

Have you also already tried this:

I have made the autotune change and rebooted the computer. I will report back.

Autotune change made no difference. Logs submitted.

Would it help to see logs from my shield when the issue occurs on the Apple TV?

I've still been experiencing the issues myself, but the frequency I'd been seeing it happen diminished significantly for a couple of months after my last post. Maybe once or twice a week.

However, within the past few weeks, it's come back with a vengeance, and it's only been happening when I start watching a show within the first couple of minutes.

Given that there's three of us running Server 2019 with the same issue, that got me thinking that maybe there's a TCP stack setting interfering with it.

I figure that most users of Channels on Windows are probably using Win 10, and they aren't reporting any issues, so I did a comparison of the TCP settings between Win 10 1909 and Server 2019 last night.

Here are the differences I noted:
Server 2019:
ECN Capability: enabled
Initial RTO: 3000
Max SYN Retransmissions: 2
Fast Open: disabled
MemoryPressureProtection: enabled
PacketCoalescingFilter: Disabled

Win 10:
ECN Capability: disabled
Initial RTO: 1000
Max SYN Retransmissions: 4
Fast Open: enabled
MemoryPressureProtection: disabled
PacketCoalescingFilter: Enabled

The one that stood out to me is the memory pressure protection. According to this article, MPP is a feature designed to protect against DDoS attacks. It can potentially misidentify an attack, and when it thinks it's under attack, it kills the existing TCP connections and drops incoming SYN requests.

I disabled it last night, and I've watched something on all three of my ATVs since then. No issues yet, so I'm hopeful this might be it.

@mhartman, give it a try by running: netsh int tcp set security mpp=disabled
If you haven't already, also set the autotuning level back to normal. That also made no difference for me.

2 Likes

@embj Nice write up and thank you! I will make this change and update the thread.

1 Like

Great find! We do have a lot of users on Windows 10 who are not experiencing issues. It's definitely plausible this setting would cause the behavior we're seeing!

So far so good. Watched two shows with no issues. Fingers crossed.

1 Like

Some bad news from me... :frowning:

I ran into the issue repeatedly this morning, even after rebooting the ATV.

Before rebooting the ATV, I disabled ECN on the server and immediately saw the stuttering again after the reboot. Then, I enabled Fast Open and rebooted the ATV. Same thing. Then, I matched the Initial RTO and Max SYN retransmits and rebooted the ATV. Same thing.

I did reboot Windows a couple of times to see if that would make a difference, and it did not. Saw the issue across two ATVs.

I re-enabled ECN, left all of the other settings I changed this morning the same, rebooted Windows and the ATV, and it's been working so far through watching three recordings on two different ATVs.

It's strange that it's so inconsistent.

Oh, and I also tried to enable the packet coalescing filter to match Win 10, and it wouldn't allow me. It said that it's unsupported on Windows Server.

I am in the same boat...experienced the issue yesterday and again this morning. I like where your head is at where some difference between W2019 and W10 is likely responsible. That said, I can't understand why the ATV exhibits the stuttering while the Shield does not when playing the same recording at the same time. I have asked the Devs about this because I'd be curious if the cache is running to 0 on the Shield too.

I ran into it yet again last night, so I dug a little deeper. When I initially looked into it the other night, I came across info about TCP templates that are configurable using PowerShell. At the time, I didn't quite understand how the templates worked, but within the templates, there are a couple of extra settings that aren't able to be set using netsh.

There are essentially two tempates; Datacenter and Internet. If the initial TCP RTT is below 10 ms, it applies the Datacenter template. If above 10 ms, it applies the Internet template. More details here.

The idea of TCP templates also exists on Win 10, but it always applies the Internet template. If you try to change the behavior to match Win Server so that it automatically picks (New-NetTransportFilter -SettingName Automatic), it tells you the feature is only available on servers.

I ran Get-NetTCPConnection -RemoteAddress {ATV IP} last night when the issue occurred, and I saw it classifying the connection to the ATV as Datacenter.

From my observation, changes made using netsh update all templates, so those settings changes being made using netsh were actually applying.

Comparing the Internet and Datacenter templates settings (besides the ones I already listed in my earlier post), the only two differences are these:
Internet
MinRto(ms): 300
DelayedAckTimeout(ms): 40
Datacenter
MinRto(ms): 20
DelayedAckTimeout(ms): 10

I also looked at Wireshark when the issue was occurring and saw endless TCP retransmits as it was trying to buffer. My Google-fu tells me that MinRto is the minimum retransmission timeout, which is the length of time that it waits to receive an ACK of the data sent. So, my guess is that the minRto was set to too small of a value, and the ATV wasn't always able to send an ACK back within 20ms. Then, it just spiraled from there and threw everything of out of sync. I'm also wondering if the times I didn't see any issue that it was using the Internet template instead??

So, I changed the MinRto to 300 to match the Internet template, downloaded TCPView on the server and used it to kill the connections to the ATV, and the buffering issue immediately corrected itself without doing anything else on the ATV.

You can change it by running:

Set-NetTCPSetting -SettingName Datacenter -MinRtoMs 300

I also re-enabled MemoryPressureProtection for the Internet profile, set the InitialRto back to 1000 for the Datacenter profile, and enabled Fast Open using netsh.

Funnily enough, googling "Apple TV minrto" directed me to this thread of users having similar buffering issues with Plex's Enhanced Video Player with ATV and Win Server. It apparently fixed buffering issues for several folks using it. That's been within the past three days that they've come to that solution. :laughing:

I've watched a couple of shows on a couple of different ATVs since then and haven't seen it so far. Given the success of users who reported it on Plex, I'm hopeful this is it.

2 Likes

Good find! I made the change on my system. Fingers crossed!

More background in this post: https://forums.plex.tv/t/enhanced-video-player-on-apple-tv-4k-creating-buffering-issues/536483/99?u=fancybits

1 Like

FYI. I fixed my network drops by swapping out my upstairs Apple TV for an Apple TV HD.

Quick update after a few weeks time...I'm happy to report that I haven't seen any stuttering issues since setting MinRto to 300.

1 Like