Okay, maybe it's not all Network Extensions apps. It seems more subtle than I thought.
Just before midnight last night, I uninstalled Little Snitch, and installed Lulu, an open source alternative that also seems to use network extensions (packet tunnel provider, app proxy provider, content-filter provider, and dns proxy) also used by Little Snitch. I set it to approve all traffic, as I had with Little Snitch. I expected the same results.
It's been 13+ hours and unaccepted connection count is still zero, system wide, despite more Censys probes and other similar remote traffic that previously triggered unaccepted connection count increases under LS.
I'll wait until later tonight, to give it a full day, uninstall Lulu and reinstall Little Snitch, but be sure to clear all of its rules and configs this time, to see if they're confounding things. In a subsequent test, I may install Cisco AnyConnect (if I can find an installer) to see if I can replicate @thully's experience.
I'll open a ticket with Little Snitch to see if they can help trouble shoot.
Maybe it could be helpful if any readers of this thread (assuming we're all mac users) that have also experienced dvr unresponsive events could run:
systemextensionsctl list
and report their results? It would be great to know what other apps may also inadvertently trigger the hangs.
Just got back home. It seems my server being down was due to the power outage - my system restarted when power was restored as I set it to, but Channels DVR didn’t start until I logged in. Is there any way to make that happen on boot in the future?
Now that the server is back up, I’ll keep an eye on the unaccepted connections and keep you posted as to whether this increases from 0 and/or the server becomes unresponsive again. I am currently running without the Cisco VPN client installed, but Renate Access enabled.
@tmm1 says below that running those doesn't affect your recordings or data:
Looking at both scripts, they only involve the removal or creation of Channel's launchd script. The --system argument ensures that Channel's launchd plist is installed in /Library/LaunchDaemons which runs channels after boot and before first login. (As opposed to in ~/Library/LaunchAgents, which only runs after your user login.)
(Alternatively, I've very infrequently experienced Channel's not fully starting itself on reboot, for some reason. The channels-dvr process is still running, but the menu bar icon never appears and http service doesn't respond. It's possible it was busy doing some longer-than-normal setup step and I was too impatient.)
As far as the hanging problem, the core issue isn't inside Channels, it's in the interaction between macOS and certain VPN/firewall apps, but affects all long-lived services running on the Mac, including channels. I've replicated it on a second machine just by installing Little Snitch, so hopefully Little Snitch will figure out a workaround and/or get Apple to fix it. (A LittleSnitch workaround won't help you, so hopefully Apple fixes the root cause.)
Ultimately, it's probably an Apple problem to fix, but Channel's could employ a workaround, either by checking unaccepted connection counts periodically to see if they exceed some threshold, or testing if it's http service is working, and restarting it if necessary, or maybe setting somaxconn before http service launch. Probably whether they do any of those depends on how many of us are affected or complaining or how long it seems it will be until Apple (or possibly developers of apps that use Apple's Network Extension Framework) ultimately fix the underlying problems.
In the mean time, if you want to continue using AnyConnect VPN without Channels hanging, you might want to run:
sudo sysctl kern.ipc.somaxconn=8192
...and then restart channels -- that increases the time before hang by 64 times, so if it used to hang once a week for you, then it'll postpone the hang by a year. (Which basically solves the problem, because Channel's updates itself much more frequently, and every update resets that count back to zero.) Unfortunately, somaxconn value will reset itself to 128 every time you reboot, so you'll need to run it again every time you reboot, unless you automate it somehow.
(I tried adding "kern.ipc.somaxconn=8192" to /private/etc/sysctl.conf, but that config file seems to be ignored by macOS now-a-days. I could add it to a new launchd plist, but I'm not sure how to guarantee it runs before channels. I could also add it to the channels launchd plist, but in the end, the somaxconn workaround just feels kinda too hacky to make permanent.)
Instead of doing all that, I just removed Little Snitch. I'll wait to see what comes up from Objective Development's (Little Snitch) efforts.
Following the instructions doesn't seem to resolve the issue - it does seem to install as a system service, but still doesn't fully respond prior to log in. I can sometimes get the admin page to start loading right after reboot, but it won't finish loading (nor will clients connect) until after I log in. One thing I notice is that once I do log in, Setup Guide Data prompts me to enter a zip code to load guide data, though my previous sources are already listed and configured properly.
As for the Cisco client - I really only installed it for my work VPN, which I only needed because my work computer stopped working a few months ago. That's also about when I remember the issue with Channels starting. Since the work PC issue has since been resolved, I can leave it uninstalled for the time being and hopefully won't have any more issues with the server becoming unresponsive. I do hope I can get the issue with reboot fixed, though, so a power outage while I'm out of town doesn't stop Channels from working once I regain power...
I think you're right. I just tested, and Channels (at least on a mac) does repeatedly crash on start up, in a loop, when run before login session is created. It looks like it dies trying to create a new menu bar icon! I emailed support and submitted diagnostics f11c99ec-1f14-4d50-8112-63a9c86fdfa0
Part of this is probably due to OS security. If you have FileVault enabled, your system's home directories will not be fully decrypted until you login. This isn't really a limitation of Channels, but rather how you chose to install it and system security measures.
Unfortunately, this is starting to show the dichotomy between server systems, and secure home systems. The line was always blurred, but as certain vendors try to be more proactively secure, it will entail more active user involvement.
@sejmann Apple requested a sysdiagnose from me. If you feel like it maybe you can submit one on feedbackassistant.apple.com and reference FB11405981
Any repro steps you can share, or any sysdiagnose you collected that captured the issue?
Please turn on NetworkExtension debug log
log config --subsystem "com.apple.networkextension" --mode level:debug,persist:debug
log config --subsystem "com.apple.networkextension" --reset <set back to default after repro>
sysctl -w net.cfil.log=6
sysctl -w net.cfil.log_port=8089
Collect sysdiagnose
@tmm1, I've submitted feedback FB11519876 with sysdiagnose after logging stuck connections, and referenced your FB.
I've also worked with Christian at obdev.at, as I think have you. After some low level logging in Little Snitch, he confirmed the problem is on Apple's side, and also has found his own server has been affected, too. Interestingly, you and Christian both passed on the same request from Apple. So, hopefully Ventura will have a fix of some kind!
Has this issue been resolved in Ventura? I still have Monterey installed and don't see any benefits to upgrading other than having this issue resolved. It still persists on the latest version of Monterey and I also have Little Snitch.
Every few days, I use the Channels app to stop and restart the DVR. If there isn't a fix, is there a way to automate restarting the server every fews hours when not in use?
If Little Snitch still causes issues, is there a way to prevent Channels DVR from auto-updating? (I apologize if my understanding of how things work is wrong. My novice-level understanding is that the problems arise when the DVR auto updates. I think I read that the pre-release version generally doesn't auto-update, but will auto-update if the release version catches up to it?). If a setting exited to make updates manual-only, maybe that would be a work around of sorts for the Little Snitch issue, so we could make sure we also reset Little Snitch at the same time we update?
I
Does this work for Little Snitch, I don't user AnyConnect VPN? I will give it a try. I'm just an average user. I tried following this thread for a solution and got lost.
So, the problem still exists in latest version Mac OS Ventura?
I'm not, yet. I've just been trying it out, and read the various warnings here in the forums, and thought I saw someone say you had to disable and reenable it, but I could have remembered that wrong.
I have now been running Little Snitch under Ventura (13.4) on an M1 Mac Mini without issue -- no random freezing of Channels DVR, and no accumulating of stuck connections (which caused the freezing) seen with netstat -anL | grep 8089, so I think the problem was fixed in Ventura.
