Channels DVR stops responding randomly on macOS

Channels DVR Server Troubleshooting
#63

Interesting. The curl didn't increment, nor reset, the netstat unaccepted connections -- still at 15.

I'm using a Ubiquity EdgeMax ER-12 router with most recent (v2.0.9-hotfix.4) firmware, with no firewall rules on the LAN side, the Mac connected directly via 1GB ethernet.

I'm also using Little Snitch on the Mac mini, which for MacOS 12 changed the way they monitor and block traffic, moving from KEXT to Apple's new NetworkExtension API, although I had the network filter turned entirely off until recently, and only used it to monitor. Also, it's always had "Allow any incoming connections " and "Allow any outgoing connections" rules for channels-dvr.

I'll update my script to include that netstat.

1 Like
#64

Interesting.. I wonder if the others here affected also are using Little Snitch.

So your output is still sitting at 15/15/128?

I was able to make those numbers go up on my own M1 mini using hping3, but I observe them resetting back to zero within a couple seconds.

$ brew install hping
$ echo; date; sudo /opt/homebrew/Cellar/hping/3.20051105/sbin/hping3 -q -c 256 -S -p 8089 --faster localhost; echo; date; netstat -anL | grep 8089; netstat -nt | grep 8089 | head -6; sleep 2; echo; date; netstat -anL | grep 8089

Fri Aug 26 14:48:05 PDT 2022
HPING localhost (utun13 127.0.0.1): S set, 40 headers + 0 data bytes
--- localhost hping statistic ---
256 packets tramitted, 0 packets received, 100% packet loss
round-trip min/avg/max = 0.0/0.0/0.0 ms

Fri Aug 26 14:48:05 PDT 2022
0/0/128        127.0.0.1.58089
128/128/128    *.8089
tcp4       0      0  127.0.0.1.8089         127.0.0.1.2722         SYN_RCVD
tcp4       0      0  127.0.0.1.8089         127.0.0.1.2721         SYN_RCVD
tcp4       0      0  127.0.0.1.8089         127.0.0.1.2720         SYN_RCVD
tcp4       0      0  127.0.0.1.8089         127.0.0.1.2719         SYN_RCVD
tcp4       0      0  127.0.0.1.8089         127.0.0.1.2718         SYN_RCVD
tcp4       0      0  127.0.0.1.8089         127.0.0.1.2717         SYN_RCVD

Fri Aug 26 14:48:07 PDT 2022
0/0/128        127.0.0.1.58089
0/0/128        *.8089
1 Like
#65

Yes, the output of netstat -anL | grep 8089 remained 15/15/128 *.8089 after running curl -s localhost:8089/status -- repeated runs of curl to localhost don't seem to increment it.

However, I just ran the netstat again, and now it's 17/17/128 *.8089 -- I'm not sure what incremented it, but another household member is now using channels in the other room via (an ethernet connected) Apple TV. And now it's 18/18/128, before even attempting your hping script.

Now I'll run: sudo hping3 -q -c 256 -S -p 8089 --faster localhost; echo; date; netstat -anL | grep 8089; netstat -nt | grep 8089 | head -6; sleep 2; echo; date; netstat -anL | grep 8089 -- it returns:

HPING localhost (utun4 127.0.0.1): S set, 40 headers + 0 data bytes

--- localhost hping statistic ---
256 packets tramitted, 0 packets received, 100% packet loss
round-trip min/avg/max = 0.0/0.0/0.0 ms

Fri Aug 26 15:05:12 PDT 2022
0/0/128        127.0.0.1.58089
128/128/128    *.8089
tcp4       0 2622136  10.0.0.11.8089         10.0.0.30.57783        ESTABLISHED
tcp4       0      0  10.0.0.11.8089         10.0.0.30.50283        ESTABLISHED
tcp4       0      0  10.0.0.11.8089         10.0.0.30.50259        ESTABLISHED
tcp4       0      0  10.0.0.11.8089         10.0.0.30.50250        ESTABLISHED
tcp4       0      0  10.0.0.11.52639        10.0.0.11.8089         TIME_WAIT
tcp4       0      0  10.0.0.11.52640        10.0.0.11.8089         TIME_WAIT

Fri Aug 26 15:05:14 PDT 2022
0/0/128        127.0.0.1.58089
18/18/128      *.8089

So, that's weird. It goes up to 128, but then resets back to 18, instead of 0.

#66

Very interesting indeed! Sounds like we're onto something, although I'm not sure what. It sounds like a bug in the macOS kernel to me.

Does systemextensionsctl list show anything besides little snitch?

#67

Nope, just little snitch. (endpoint_security one is their tap into the Berkeley packet filter, which I only turned on last week.)

$ systemextensionsctl list
2 extension(s)
--- com.apple.system_extension.network_extension
enabled	active	teamID	bundleID (version)	name	[state]
*	*	MLZF7K7B5R	at.obdev.littlesnitch.networkextension (5.4.1/6256)	Little Snitch Network Extension	[activated enabled]
--- com.apple.system_extension.endpoint_security
enabled	active	teamID	bundleID (version)	name	[state]
*	*	MLZF7K7B5R	at.obdev.littlesnitch.endpointsecurity (5.4.1/6256)	Little Snitch Endpoint Security	[activated enabled]

I wonder if @thully is also using Little Snitch. I can remove Little Snitch entirely shortly, to help rule it out or in, I guess, at least to see if it affects unaccepted packets.

#68

Now that we have a signal to look at, here's a few things I would try:

  1. restart channels-dvr so the numbers reset to 0/0/128
  2. turn off remote port forwarding so no traffic from the internet is coming through
  3. wait a day and see if the qlen is still 0 or not

then try the same but with

  1. disable little snitch
#69

Here's my output:
sysctl kern.ipc.somaxconn
kern.ipc.somaxconn: 128

netstat -anL | grep 8089

0/0/128        127.0.0.1.58089        
19/19/128      *.8089

I'm using the Arris BGW210 provided with my AT&T fiber connection, as well as an old Time Capsule with routing disabled (behaving as a switch) which my Mac is connected to by Ethernet. I'm not using Little Snitch.

When you say disable remote port forwarding, do you mean on my router or in Channels (i.e. disabling remote DVR)?

#70

Okay, I'll be recording the netstat -anL minutely. Before your reply, I'd jumped the gun and uninstalled Little Snitch, so I guess my first testing period will be without Little Snitch in the mix. I've turned off Remote DVR. Unfortunately, we won't be watching any TV this weekend, so we won't be exercising channels in the usual way until Monday, although I can leave an Apple TV client playing continuously. Also installed v2022.08.26.2245 with improved macOS diagnostics.

Okay, @thully's at 19 -- definitely on to something with this! And no Little Snitch, so that's probably not a root cause.

I can maybe add a WAN port forward from :8089 to :58089 and see if any external traffic is causing the unaccepted packet increments -- I noticed @thully had some external Censys traffic in the hours before his last hang -- I wonder if they're doing some kind of error response fingerprinting involving partial packets. -- oh, 58089 is localhost only. I can't do that. Oh well.

@thully : Yes, you're exactly right, turning off "Remote DVR" shuts off the NATPMP port forwarding in your Airport Time Machine to channels, so external traffic won't reach Channel's DVR. Although, you won't be able to connect remotely, of course, with that turned off, in case you're traveling.

1 Like
#71

The :58089 listener is only on localhost, so it won't accept packets from elsewhere. It's not setup the same way with SSL support.

You could do a simple socat listener on a random port though, expose that and see if the qlen ticks up.

Another thing that may make a difference is bumping the global somaxconn limit. Although when I tried, netstat -anL still reported a maxqlen of 128.

sudo sysctl kern.ipc.somaxconn=8192

#72

Same here

#73

Another user who reported similar issues over support is also using Little Snitch.

@thully Can you run this and post the output:

systemextensionsctl list

#74

For the weekend, I'm trying

socat TCP4-LISTEN:1025,fork STDOUT

much later corrected to:

socat TCP4-LISTEN:1025,fork,backlog=8192 STDOUT

and set up a TCP port forward from WAN 8089 -> Mac mini 1025
no TLS configured of course, as well -- although, that shouldn't matter, I guess.

Interestingly, socat uses a max of 5 instead of 128, by default, as in 0/0/5. I'd have to dig in to it's man page to reconfigure, if possible. (This max seems to be configured by launching apps instead of globally? because netstat -anL shows several different max queue lengths for different ports: max 5, 32 and 128 -- so maybe golang has a way to set it, too?)

I'm also running Jellyfin on exposed to WAN,and it's also probed frequently, so I'll monitor it's unaccepted count.

1 Like
#75 
LISTEN option group

Options specific to listening sockets.

backlog=<count>
Sets the backlog value passed with the listen() system call to <count> [int]. Default is 5.
#76

Looks like if you restart the DVR after bumping somaxconn, it will see and use the new limit:

$ sysctl kern.ipc.somaxconn
kern.ipc.somaxconn: 8192

$ netstat -anL | grep 8089
0/0/8192       127.0.0.1.58089
0/0/8192       *.8089
1 Like
#77

Oh, the 8192 took affect only after a channels-dvr restart -- that makes sense.

$ date;netstat -anL | grep 8089
Fri Aug 26 16:57:54 PDT 2022
0/0/8192 127.0.0.1.58089
0/0/8192 *.8089

Oops, you beat me to it.

#78

Here's what I have:
1 extension(s)

--- com.apple.system_extension.network_extension

enabled active teamID bundleID (version) name [state]

    • DE8Y96K9QP com.cisco.anyconnect.macos.acsockext (4.10.03104/4.10.03104) Cisco AnyConnect Socket Filter Extension [activated enabled]

That's a VPN client - though I haven't connected to the VPN in several weeks.

#79

Interesting. I wonder if everyone affected is using the Network Extension API.

@tmm1 -- did you add systemextensionsctl list to macOS diagnostics in v2022.08.26.2245?

#80

Yes I did

2 Likes
#81

After disabling remote access/port forwarding and restarting/updating Channels, my netstat output reset to 0/0/128. However, about an hour, it was back up to 4/4/128.

Given that, I decided to try uninstalling the Cisco VPN software and restarting my Mac to see if that has an impact. Will see if the number goes up again or stays at 0...

2 Likes
#82

Any observations through the weekend?