Uh, sorry, this is long and rambly, but: With Little Snitch uninstalled, unaccepted connections count continuously remained 0 for port 8089, over the course of the initial 48 hours since my last post -- I checked once every 60 seconds.
At about half way through those 48 hours, I turned on Remote DVR port forwarding, and it also had no affect on unaccepted connection counts over the next 24 hours. They remained 0.
Last night, (about 18 hours ago) I reinstalled little snitch. Unaccepted connections climbed to 5 after a six hours. Then, 12 hours ago, I upgraded channels DVR to the then latest prerelease 2022.08.29.0732, which reset the count to 0. In the last 12 hours it's climbed to 3, although two minutes ago, it jumped from 3 to 5.
And it didn't just affect channels -- several services (homebridge, jellyfin, and an rtsp relay) also have unaccepted connections tick up since little snitch was reinstalled. They had remained 0 while little snitch was uninstalled.
Looking at the time of the last jump, between 2:59p and 3:00p,.
Mon Aug 29 14:59:11 PDT 2022
0/0/8192 127.0.0.1.58089
3/3/8192 *.8089
Mon Aug 29 15:00:11 PDT 2022
0/0/8192 127.0.0.1.58089
5/5/8192 *.8089
Between those times, this was the channels http log:
2022/08/29 14:59:12.097247 [HTTP] | 200 | 99.639ms | 10.0.0.123 | GET "/devices/10A22AE4/status/2-1/debug"
2022/08/29 14:59:14.190506 [HTTP] | 200 | 85.168333ms | 10.0.0.123 | GET "/devices/10A22AE4/status/2-1/debug"
2022/08/29 14:59:16.272905 [HTTP] | 200 | 67.091625ms | 10.0.0.123 | GET "/devices/10A22AE4/status/2-1/debug"
2022/08/29 14:59:18.355584 [HTTP] | 200 | 64.539416ms | 10.0.0.123 | GET "/devices/10A22AE4/status/2-1/debug"
2022/08/29 14:59:20.419733 [HTTP] | 200 | 56.198458ms | 10.0.0.123 | GET "/devices/10A22AE4/status/2-1/debug"
2022/08/29 14:59:22.490728 [HTTP] | 200 | 60.866416ms | 10.0.0.123 | GET "/devices/10A22AE4/status/2-1/debug"
2022/08/29 14:59:24.556013 [HTTP] | 200 | 50.933875ms | 10.0.0.123 | GET "/devices/10A22AE4/status/2-1/debug"
2022/08/29 14:59:26.664447 [HTTP] | 200 | 93.738458ms | 10.0.0.123 | GET "/devices/10A22AE4/status/2-1/debug"
2022/08/29 14:59:28.765248 [HTTP] | 200 | 78.621041ms | 10.0.0.123 | GET "/devices/10A22AE4/status/2-1/debug"
2022/08/29 14:59:30.871245 [HTTP] | 200 | 83.696458ms | 10.0.0.123 | GET "/devices/10A22AE4/status/2-1/debug"
2022/08/29 14:59:32.988058 [HTTP] | 200 | 99.479167ms | 10.0.0.123 | GET "/devices/10A22AE4/status/2-1/debug"
2022/08/29 14:59:35.049449 [HTTP] | 200 | 47.629375ms | 10.0.0.123 | GET "/devices/10A22AE4/status/2-1/debug"
2022/08/29 14:59:37.110325 [HTTP] | 200 | 50.7985ms | 10.0.0.123 | GET "/devices/10A22AE4/status/2-1/debug"
2022/08/29 14:59:39.169727 [HTTP] | 200 | 49.5875ms | 10.0.0.123 | GET "/devices/10A22AE4/status/2-1/debug"
2022/08/29 14:59:41.296000 [HTTP] | 200 | 111.188834ms | 10.0.0.123 | GET "/devices/10A22AE4/status/2-1/debug"
2022/08/29 14:59:43.358467 [HTTP] | 200 | 45.947042ms | 10.0.0.123 | GET "/devices/10A22AE4/status/2-1/debug"
2022/08/29 14:59:45.426466 [HTTP] | 200 | 46.895542ms | 10.0.0.123 | GET "/devices/10A22AE4/status/2-1/debug"
2022/08/29 14:59:47.561894 [HTTP] | 200 | 116.545625ms | 10.0.0.123 | GET "/devices/10A22AE4/status/2-1/debug"
2022/08/29 14:59:49.528301 [HTTP] | 200 | 127.917µs | 167.94.138.62 | GET "/"
2022/08/29 14:59:49.625967 [HTTP] | 200 | 34.993083ms | 10.0.0.123 | GET "/devices/10A22AE4/status/2-1/debug"
2022/08/29 14:59:49.638067 [HTTP] | 200 | 81.709µs | 167.94.138.62 | GET "/"
2022/08/29 14:59:49.747911 [HTTP] | 200 | 97.083µs | 167.94.138.62 | PRI "*"
2022/08/29 14:59:49.860210 [HTTP] | 200 | 110.75µs | 167.94.138.62 | GET "/favicon.ico"
2022/08/29 14:59:51.692148 [HTTP] | 200 | 47.193417ms | 10.0.0.123 | GET "/devices/10A22AE4/status/2-1/debug"
2022/08/29 14:59:53.757780 [HTTP] | 200 | 46.180542ms | 10.0.0.123 | GET "/devices/10A22AE4/status/2-1/debug"
2022/08/29 14:59:55.824103 [HTTP] | 200 | 57.614541ms | 10.0.0.123 | GET "/devices/10A22AE4/status/2-1/debug"
2022/08/29 14:59:57.898898 [HTTP] | 200 | 66.926041ms | 10.0.0.123 | GET "/devices/10A22AE4/status/2-1/debug"
2022/08/29 14:59:59.983117 [HTTP] | 200 | 68.110667ms | 10.0.0.123 | GET "/devices/10A22AE4/status/2-1/debug"
2022/08/29 15:00:02.042379 [HTTP] | 200 | 56.017167ms | 10.0.0.123 | GET "/devices/10A22AE4/status/2-1/debug"
2022/08/29 15:00:04.082324 [HTTP] | 200 | 31.664459ms | 10.0.0.123 | GET "/devices/10A22AE4/status/2-1/debug"
2022/08/29 15:00:06.159674 [HTTP] | 200 | 69.937125ms | 10.0.0.123 | GET "/devices/10A22AE4/status/2-1/debug"
2022/08/29 15:00:08.225525 [HTTP] | 200 | 52.189833ms | 10.0.0.123 | GET "/devices/10A22AE4/status/2-1/debug"
2022/08/29 15:00:08.433785 [HTTP] | 200 | 4.885585083s | 10.0.0.123 | GET "/tmsimg/assets/p10158345_b_h9_aa.jpg?w=360"
2022/08/29 15:00:10.323723 [HTTP] | 200 | 81.0325ms | 10.0.0.123 | GET "/devices/10A22AE4/status/2-1/debug"
Coincidentally, the 167.94.* ip address is Censys again remotely probing channels.
More interestingly, the prior jump happened with no corresponding http traffic logged
Mon Aug 29 07:02:38 PDT 2022
0/0/8192 127.0.0.1.58089
2/2/8192 *.8089
Mon Aug 29 07:03:38 PDT 2022
0/0/8192 127.0.0.1.58089
3/3/8192 *.8089
2022/08/29 04:20:23.066568 [HTTP] | 200 | 1.531584ms | 185.220.101.58 | GET "/"
2022/08/29 04:20:24.611134 [HTTP] | 200 | 378.208µs | 185.220.101.161 | GET "/favicon.ico"
2022/08/29 13:08:13.300542 [HTTP] | 200 | 205.201167ms | 10.0.0.11 | GET "/dvr/files"
2022/08/29 13:08:13.521647 [HTTP] | 200 | 150.601958ms | 10.0.0.11 | DELETE "/dvr/files/15103"
However, I did also capture a tcpdump of port 8089 over the last 12 hours,nas well. I see this in that time period:
07:02:47.861463 IP ns3181291.ip-151-106-40.eu.57898 > macmini.8089: Flags [S], seq 4279450726, win 1024, length 0
E..(_).....o.j(.
....*....<f....P....m........
07:02:48.034109 IP ns3181291.ip-151-106-40.eu.57898 > macmini.8089: Flags [R], seq 4279450727, win 0, length 0
E..(%n@.j.!+.j(.
....*....<g..<gP...l.........
07:02:48.036569 IP ns3181291.ip-151-106-40.eu.57898 > macmini.8089: Flags [R], seq 4279450727, win 1200, length 0
E..(_(.....p.j(.
....*....<g....P.............
So it's some kind of non-http TCP traffic -- I haven't yet tried to understand it.
And the prior jump, also not in the http logs:
Mon Aug 29 04:56:29 PDT 2022
0/0/8192 127.0.0.1.58089
1/1/8192 *.8089
Mon Aug 29 04:57:29 PDT 2022
0/0/8192 127.0.0.1.58089
2/2/8192 *.8089
but in tcpdump I see:
04:56:59.567501 IP 89.248.171.133.54116 > 10.0.0.11.8089: Flags [S], seq 587785294, win 65535, length 0
E..(.1......Y...
....d..#..N....P.............
04:56:59.705773 IP 89.248.171.133.54116 > 10.0.0.11.8089: Flags [R], seq 587785295, win 0, length 0
E..(..@...=HY...
....d..#..O....P.............
Some more non-http traffic. There's no other surrounding traffic for quite a while, so for whatever reason, those packets cause the unaccepted count to tick up, but, seemingly, only while little snitch is installed.
I think I may turn off Remote DVR again, and figure out what local packets, if any, cause the count to tick up.
Also, I'm still not sure I understand what the listen queue behavior should be. Does the fact that the unaccepted connection count, and unaccepted incomplete connection count rise in tandem mean simply that incomplete connections tally to both totals -- is the first inclusive of the second, and that there simply weren't unaccepted connections for reasons other than incompleteness? And are those counts meant to reset after a period of time, or do they persist the duration of the socket listening? I mean, clearly they do persist in our case, but are they supposed to? Are the incomplete connections just waiting to be completed, and thus consuming "slots" in the max listening connections queue? Is Apple's Network Extension API just dropping certain (rejected/malformed?) TCP SYN packets, leaving the connection in an incomplete state, and for some reason, things aren't timing out properly?
I also want to check if I can install another Network Extension API using app, instead of Little Snitch and see if the problem persists.