Interruptions during recordings and streaming with HDHR Prime and Channels DVR Server

[Milky]

The HDHR is connected via a tested (and replaced to rule out) CAT6E cable that is less than 8feet. It connects to a Brocade ICX-7250 24P switch (24 1gig ports, 2 10 gig ports). One 10G port goes to the server running Proxmox (10 vms, 2 containers). Yesterday I set up a dedicated VM on said server for Channels DVR server to rule out an issue with running in Docker. Server load is typically under 5% cpu, and very little traffic or disk activity. It has 384 gigs of ECC ram, so plenty of headroom there as well. Both the HDHR and the VM are on the same VLAN.

On my phone, I am able to run 2 apps at once. In the split screen, I used channels DVR and HDHR's android app. Viewing the same program, the channels DVR will drop, while the HDHR will never hiccup or anything.

From the other thread, which I think I have resolved:

@tmm1

is your signal amplified?

No, unless there is an amp in the Verizon FiOS fiber ONT. My setup is: FiOS fiber into ONT, Coax to 5 way (1 in, 4 out, 7.5db) splitter. both cables from ONT to splitter, and splitter to HDHR prime are under 3 feet. I tried adding a second splitter to drop signal dbs, which it did, but recording interruptions got worse. It would also say 'Card OOB Lock weak' with the second splitter. Now, it says success with the high signal strength. The other 3 coax cables go to my 2 verizon boxes, and 1 actiontec moca bridge. Zero issues with any of those.

I'm willing to test pretty much anything to rule it out beyond what I already have, any suggestions are appreciated.

Here's an example of an interrupted recording from last night:

Channels log:

2024/01/10 21:00:00.001646 [DVR] Starting job 1704938400-32 Guy's Grocery Games on ch=[664 6102]
2024/01/10 21:00:00.002282 [DVR] Waiting 59m59.997718801s until next job 1704942000-32 Guy's Grocery Games
2024/01/10 21:00:00.009491 [TNR] Closed connection to 1313B64C/1 for ch664 Food Network HD
2024/01/10 21:00:00.009580 [TNR] Closed connection to 1313B64C/0 for ch511 CW WPIX-DT
2024/01/10 21:00:00.037855 [SNR] Signal statistics for "TV/Guy's Grocery Games/Guy's Grocery Games S15E10 2017-12-17 Last Judge Standing Tourn 2024-01-10-2000.mpg": ss=100% snq=99%-100% seq=100% bps=13067529,2681632-18424000 pps=1118,230-1577
2024/01/10 21:00:00.037942 [SNR] Signal statistics for "TV/Whose Line Is It Anyway/Whose Line Is It Anyway S13E08 2024-01-09 Mark Ballas 2024-01-10-2030.mpg": ss=100% snq=99%,95%-100% seq=99%,0%-100% bps=9004828,0-14259424 pps=771,0-1221
2024/01/10 21:00:00.052143 [SNR] Buffer statistics for "TV/Guy's Grocery Games/Guy's Grocery Games S15E10 2017-12-17 Last Judge Standing Tourn 2024-01-10-2000.mpg": buf=0%-2% drop=0%
2024/01/10 21:00:00.057128 [SNR] Buffer statistics for "TV/Whose Line Is It Anyway/Whose Line Is It Anyway S13E08 2024-01-09 Mark Ballas 2024-01-10-2030.mpg": buf=0%-3% drop=0%
2024/01/10 21:00:00.386409 [DVR] Finished job 1704936600-10 Whose Line Is It Anyway?
2024/01/10 21:00:00.403521 [DVR] Waiting 59m59.596480299s until next job 1704942000-32 Guy's Grocery Games
2024/01/10 21:00:00.414701 [DVR] Processing file-860: TV/Whose Line Is It Anyway/Whose Line Is It Anyway S13E08 2024-01-09 Mark Ballas 2024-01-10-2030.mpg
2024/01/10 21:00:00.508260 [TNR] Opened connection to 1313B64C/0 for ch664 Food Network HD
2024/01/10 21:00:01.009526 [DVR] Recording for job 1704938400-32 from 1313B64C ch664 into "TV/Guy's Grocery Games/Guy's Grocery Games S33E00 Relay Winners vs Judges 2024-01-10-2100.mpg" for 59m59.997611524s
2024/01/10 21:00:01.064220 [IDX] Generating video index for job 1704938400-32
2024/01/10 21:00:03.115906 [DVR] Finished job 1704934800-32 Guy's Grocery Games
2024/01/10 21:00:03.132555 [DVR] Waiting 59m56.867445968s until next job 1704942000-32 Guy's Grocery Games
2024/01/10 21:00:03.193570 [DVR] Processing file-859: TV/Guy's Grocery Games/Guy's Grocery Games S15E10 2017-12-17 Last Judge Standing Tourn 2024-01-10-2000.mpg
2024/01/10 21:00:03.223383 [DVR] Running commercial detection on file 860 (TV/Whose Line Is It Anyway/Whose Line Is It Anyway S13E08 2024-01-09 Mark Ballas 2024-01-10-2030.mpg)
2024/01/10 21:01:31.496654 [DVR] Commercial detection for Whose Line Is It Anyway S13E08 2024-01-09 Mark Ballas 2024-01-10-2030.mpg finished with 8 markers in 1m28.301338488s (12 threads).
2024/01/10 21:01:51.757536 [DVR] Running commercial detection on file 859 (TV/Guy's Grocery Games/Guy's Grocery Games S15E10 2017-12-17 Last Judge Standing Tourn 2024-01-10-2000.mpg)
2024/01/10 21:05:15.268779 [DVR] Commercial detection for Guy's Grocery Games S15E10 2017-12-17 Last Judge Standing Tourn 2024-01-10-2000.mpg finished with 10 markers in 3m23.516290533s (12 threads).
2024/01/10 21:07:47.568997 [DVR] Fetched guide data for XMLTV-Samsung in 1s
2024/01/10 21:07:49.905417 [DVR] Indexed 1747 airings into XMLTV-Samsung (236 channels over 11h0m0s) + 341 skipped [2s index]
2024/01/10 21:07:49.925008 [DVR]   pruned 711 replaced airings in 18ms.
2024/01/10 21:07:50.057440 [DVR] Fetched guide data for XMLTV-Plex in 130ms
2024/01/10 21:08:05.416830 [DVR] Indexed 12901 airings into XMLTV-Plex (432 channels over 7318h55m36s) + 551 skipped [14s index]
2024/01/10 21:08:05.455992 [DVR]   pruned 666 replaced airings in 39ms.
2024/01/10 21:08:06.068875 [DVR] Rule 'American Restoration' (28) found 3 new airings
2024/01/10 21:08:06.071591 [DVR]   queued recording job for American Restoration/S1E16 @ Thu Jan 11 7:00:00PM on ch=[9617] (1705017600-28)
2024/01/10 21:08:06.074073 [DVR]   queued recording job for American Restoration/S1E13 @ Thu Jan 11 7:25:50PM on ch=[9617] (1705019150-28)
2024/01/10 21:08:06.076546 [DVR]   queued recording job for American Restoration/S3E6 @ Thu Jan 11 7:51:30PM on ch=[9617] (1705020690-28)
2024/01/10 21:08:06.221621 [DVR] Waiting 51m53.778379506s until next job 1704942000-32 Guy's Grocery Games
2024/01/10 21:13:19.772545 [DVR] Starting job 1704939199-ch10282 Hell's Kitchen on ch=[10282]
2024/01/10 21:13:19.772809 [DVR] Waiting 46m40.227191609s until next job 1704942000-32 Guy's Grocery Games
2024/01/10 21:13:19.773493 [TNR] Sharing existing connection to M3U-Samsung for ch10282 Hell's Kitchen (clients=2, len=0)
2024/01/10 21:13:19.780358 [DVR] Recording for job 1704939199-ch10282 from M3U-Samsung ch10282 into "TV/Hell's Kitchen/Hell's Kitchen Season 6 5 Chefs Compete 2024-01-10-2113.mpg" for 31m13.227130993s
2024/01/10 21:13:19.823056 [IDX] Generating video index for job 1704939199-ch10282
2024/01/10 21:13:31.590656 [DVR] Waiting 31m1.409352666s until next job 1704941073-ch10282 Hell's Kitchen
2024/01/10 21:13:32.189834 [SNR] Rewriter statistics for 10.1.10.57 (Living Room TV) for ch10282 Hell's Kitchen: discontinuity_detected=0 transport_errors=0 saw_pcr=true saw_pmt=true highest_pts=2048.096733
2024/01/10 21:13:32.190510 [SNR] Buffer statistics for 10.1.10.57 (Living Room TV) for ch10282 Hell's Kitchen: buf=0% drop=0%
2024/01/10 21:13:32.190648 [SNR] Streaming statistics for 10.1.10.57 (Living Room TV) for ch10282 Hell's Kitchen: timeouts=0 segment_timeouts=0 playlist_timeouts=0
2024/01/10 21:13:36.366374 [DVR] Waiting 30m56.633628038s until next job 1704941073-ch10282 Hell's Kitchen
2024/01/10 21:19:19.950635 [TNR] Cancelling stream 1313B64C/0 ch664 after no data was received for 6s
2024/01/10 21:19:19.965001 [TNR] Closed connection to 1313B64C/0 for ch664 Food Network HD
2024/01/10 21:19:19.986238 [SNR] Signal statistics for "TV/Guy's Grocery Games/Guy's Grocery Games S33E00 Relay Winners vs Judges 2024-01-10-2100.mpg": ss=100% snq=99%,98%-100% seq=99%,0%-100% bps=13406991,0-18159296 pps=1141,0-1555
2024/01/10 21:19:20.002852 [SNR] Buffer statistics for "TV/Guy's Grocery Games/Guy's Grocery Games S33E00 Relay Winners vs Judges 2024-01-10-2100.mpg": buf=0%-4% drop=0%
2024/01/10 21:19:20.313864 [DVR] Job 1704938400-32 Guy's Grocery Games ended prematurely: 40m39.686142751s
2024/01/10 21:19:20.332391 [DVR] Starting job 1704938400-32 Guy's Grocery Games on ch=[664 6102]
2024/01/10 21:19:20.332554 [DVR] Waiting 25m12.667446642s until next job 1704941073-ch10282 Hell's Kitchen
2024/01/10 21:19:20.879069 [TNR] Opened connection to 1313B64C/0 for ch664 Food Network HD
2024/01/10 21:19:20.881304 [DVR] Recording for job 1704938400-32 from 1313B64C ch664 into "TV/Guy's Grocery Games/Guy's Grocery Games S33E00 Relay Winners vs Judges 2024-01-10-2100.mpg" for 40m39.667390365s
2024/01/10 21:44:33.005771 [DVR] Starting job 1704941073-ch10282 Hell's Kitchen on ch=[10282]
2024/01/10 21:44:33.006310 [DVR] Waiting 15m26.993690813s until next job 1704942000-32 Guy's Grocery Games
2024/01/10 21:44:33.007460 [TNR] Sharing existing connection to M3U-Samsung for ch10282 Hell's Kitchen (clients=2, len=0)
2024/01/10 21:44:33.012742 [DVR] Recording for job 1704941073-ch10282 from M3U-Samsung ch10282 into "TV/Hell's Kitchen/Hell's Kitchen Season 6 4 Chefs Compete 2024-01-10-2144.mpg" for 51m53.993589069s
2024/01/10 21:44:33.061149 [IDX] Generating video index for job 1704941073-ch10282
2024/01/10 21:44:54.122206 [SNR] Buffer statistics for "TV/Hell's Kitchen/Hell's Kitchen Season 6 5 Chefs Compete 2024-01-10-2113.mpg": buf=0% drop=0%
2024/01/10 21:44:54.122292 [SNR] Streaming statistics for "TV/Hell's Kitchen/Hell's Kitchen Season 6 5 Chefs Compete 2024-01-10-2113.mpg": timeouts=0 segment_timeouts=0 playlist_timeouts=0
2024/01/10 21:44:54.133595 [MTS] Statistics for "TV/Hell's Kitchen/Hell's Kitchen Season 6 5 Chefs Compete 2024-01-10-2113.mpg": discontinuity_detected=0 transport_errors=0 saw_pcr=true saw_pmt=true highest_pts=1915.968733
2024/01/10 21:44:54.237288 [DVR] Finished job 1704939199-ch10282 Hell's Kitchen
2024/01/10 21:44:54.254264 [DVR] Waiting 15m5.745736953s until next job 1704942000-32 Guy's Grocery Games
2024/01/10 21:44:54.266062 [DVR] Processing file-862: TV/Hell's Kitchen/Hell's Kitchen Season 6 5 Chefs Compete 2024-01-10-2113.mpg
2024/01/10 21:44:55.911376 [DVR] Running commercial detection on file 862 (TV/Hell's Kitchen/Hell's Kitchen Season 6 5 Chefs Compete 2024-01-10-2113.mpg)
2024/01/10 21:52:11.618144 [DVR] Commercial detection for Hell's Kitchen Season 6 5 Chefs Compete 2024-01-10-2113.mpg finished with 4 markers in 7m15.716938758s (12 threads).
2024/01/10 22:00:00.001554 [DVR] Starting job 1704942000-32 Guy's Grocery Games on ch=[664 6102]
2024/01/10 22:00:00.001741 [DVR] Waiting 25m41.998259583s until next job 1704943542-19 Forged in Fire
2024/01/10 22:00:00.005641 [TNR] Closed connection to 1313B64C/0 for ch664 Food Network HD
2024/01/10 22:00:00.021087 [SNR] Signal statistics for "TV/Guy's Grocery Games/Guy's Grocery Games S33E00 Relay Winners vs Judges 2024-01-10-2100.mpg": ss=100% snq=99%,97%-100% seq=100% bps=12885413,5100064-18190880 pps=1103,437-1557
2024/01/10 22:00:00.034876 [SNR] Buffer statistics for "TV/Guy's Grocery Games/Guy's Grocery Games S33E00 Relay Winners vs Judges 2024-01-10-2100.mpg": buf=0% drop=0%
2024/01/10 22:00:00.165204 [DVR] Finished job 1704938400-32 Guy's Grocery Games
2024/01/10 22:00:00.185960 [DVR] Waiting 25m41.814041248s until next job 1704943542-19 Forged in Fire
2024/01/10 22:00:00.192355 [DVR] Processing file-861: TV/Guy's Grocery Games/Guy's Grocery Games S33E00 Relay Winners vs Judges 2024-01-10-2100.mpg
2024/01/10 22:00:00.513166 [TNR] Opened connection to 1313B64C/0 for ch664 Food Network HD
2024/01/10 22:00:00.516463 [DVR] Recording for job 1704942000-32 from 1313B64C ch664 into "TV/Guy's Grocery Games/Guy's Grocery Games S22E07 2019-11-06 Cheat Day 2024-01-10-2200.mpg" for 59m59.998213488s
2024/01/10 22:00:00.573461 [IDX] Generating video index for job 1704942000-32
2024/01/10 22:00:00.685190 [IDX] Generating video index for file-861: TV/Guy's Grocery Games/Guy's Grocery Games S33E00 Relay Winners vs Judges 2024-01-10-2100.mpg
2024/01/10 22:00:00.702719 [DVR] Running commercial detection on file 861 (TV/Guy's Grocery Games/Guy's Grocery Games S33E00 Relay Winners vs Judges 2024-01-10-2100.mpg)
2024/01/10 22:00:55.627769 [IDX] Finished video index generation for file-861 in 54s
2024/01/10 22:03:19.397881 [DVR] Commercial detection for Guy's Grocery Games S33E00 Relay Winners vs Judges 2024-01-10-2100.mpg finished with 10 markers in 3m18.700183899s (12 threads).
2024/01/10 22:08:06.832480 [DVR] Fetched guide data for XMLTV-Plex in 1s
2024/01/10 22:08:21.747650 [DVR] Indexed 12349 airings into XMLTV-Plex (432 channels over 7318h55m36s) + 1103 skipped [14s index]
2024/01/10 22:08:21.783083 [DVR]   pruned 552 replaced airings in 34ms.
2024/01/10 22:08:22.490204 [DVR] Waiting 17m19.509796741s until next job 1704943542-19 Forged in Fire

HDHR Log:

20240111-01:59:59 CableCARD: tuner0 664 Food Network HD (auto:597MHz-629) access = subscribed
20240111-02:19:19 Tuner: tuner0 http stream ended (remote closed)
20240111-02:19:19 Tuner: tuner0 tuning 664 Food Network HD (auto:597MHz-629)
20240111-02:19:20 Tuner: tuner0 streaming http to 10.69.10.22:50938
20240111-02:19:20 CableCARD: tuner0 664 Food Network HD (auto:597MHz-629) access = subscribed
20240111-02:59:59 Tuner: tuner0 http stream ended (remote closed)

[tmm1]

Can you try removing as many splitters as possible to see if there is any improvement?

[Milky]

@tmm1

Can you try removing as many splitters as possible to see if there is any improvement?

There's only one splitter. I can split the feed into a 2 way first and have one leg feed the hdhr, and another feed the other splitter, but signal strength doesn't look like an issue? And why would the hdhr app stay connected while channels drops if it were a signal issue?

[Milky]

[chDVRuser]

Here are the specs for the Prime Signal Strength.
Start viewing a channel with your HDHR app or Channels and look at the HDHR web UI for the tuner being used (not the CableCARD Status). On a Prime it will display the dBmV level.

Signal Strength
100% = 0dBmV = -48.75dBm and a change of 5% is a 3dBmV difference, so every 1% change is equivilent to 0.6dBmV
95% = -3dBmV = -51.75dBm
90% = -6dBmV = -54.75dBm

A signal strength of 90% (-6dBmV) or higher is recommended for QAM operation, 80% (-12dBmV) minimum.
US Cable specs are 0dBmV +/- 12dBmV (80-120%)

[tmm1]

@Milky

Viewing the same program, the channels DVR will drop, while the HDHR will never hiccup or anything.

Hmm was this with tuner sharing on or off?

[Milky]

@chDVRuser

Start viewing a channel with your HDHR app or Channels and look at the HDHR web UI for the tuner being used (not the CableCARD Status). On a Prime it will display the dBmV level.

@tmm1

Hmm was this with tuner sharing on or off?

I'm pretty sure it was on then, but it is definitely on now. (I've tried so much)

[Milky]

I split the feed from the ONT 2 ways. One feed to HDHR prime, other feed to the other splitter.

Watching hockey last night live, it disconnected again. A show that was recording at the same time did not have an interruption. Here's the relevant log lines.

2024/01/11 21:23:56.610256 [TNR] Opened connection to 1313B64C/0 for ch579 MSG 2 HD
2024/01/11 21:35:58.018635 [TNR] Cancelling stream 1313B64C/0 ch579 after no data was received for 6s
2024/01/11 21:35:58.034064 [TNR] Closed connection to 1313B64C/0 for ch579 MSG 2 HD
2024/01/11 21:35:58.034148 [SNR] Statistics for ch579 MSG 2 HD: ss=100% snq=100% seq=100% bps=18193683,3349408-18460096 pps=1547,0-1589
2024/01/11 21:35:58.034426 [SNR] Buffer statistics for 10.1.10.40 (S23 Ultra) for ch579 MSG 2 HD: buf=0% drop=0%

20240112-02:23:55 CableCARD: tuner0 579 MSG 2 HD (auto:147MHz-760) access = subscribed
20240112-02:29:59 Tuner: tuner2 http stream ended (remote closed)
20240112-02:29:59 Tuner: tuner1 tuning 664 Food Network HD (auto:597MHz-629)
20240112-02:30:00 Tuner: tuner1 streaming http to 10.69.10.22:35972
20240112-02:30:00 CableCARD: tuner1 664 Food Network HD (auto:597MHz-629) access = subscribed
20240112-02:35:57 Tuner: tuner0 http stream ended (remote closed)

It also dropped while recording a show later.

2024/01/11 22:30:00.002958 [DVR] Starting job 1705030200-30 Burgers, Brew & 'Que on ch=[663 6109]
2024/01/11 22:30:00.004001 [DVR] Waiting 1h29m59.99599987s until next job 1705035600-28 American Restoration
2024/01/11 22:30:00.006928 [TNR] Closed connection to 1313B64C/1 for ch663 Cooking Channel
2024/01/11 22:30:00.027307 [SNR] Signal statistics for "TV/Burgers, Brew & 'Que/Burgers, Brew & 'Que S05E08 2019-03-19 OG Flavor 2024-01-11-2200.mpg": ss=100% snq=100% seq=100% bps=13183623,6204000-18428512 pps=1128,533-1578
2024/01/11 22:30:00.043582 [SNR] Buffer statistics for "TV/Burgers, Brew & 'Que/Burgers, Brew & 'Que S05E08 2019-03-19 OG Flavor 2024-01-11-2200.mpg": buf=0%-1% drop=0%
2024/01/11 22:30:00.363434 [DVR] Finished job 1705028400-30 Burgers, Brew & 'Que
2024/01/11 22:30:00.385394 [DVR] Waiting 1h29m59.61460697s until next job 1705035600-28 American Restoration
2024/01/11 22:30:00.392124 [DVR] Processing file-892: TV/Burgers, Brew & 'Que/Burgers, Brew & 'Que S05E08 2019-03-19 OG Flavor 2024-01-11-2200.mpg
2024/01/11 22:30:00.528921 [TNR] Opened connection to 1313B64C/1 for ch663 Cooking Channel
2024/01/11 22:30:00.541370 [DVR] Recording for job 1705030200-30 from 1313B64C ch663 into "TV/Burgers, Brew & 'Que/Burgers, Brew & 'Que 2019-12-17 Champion Burger Knockouts 2024-01-11-2230.mpg" for 29m59.99589844s
2024/01/11 22:30:00.590986 [IDX] Generating video index for job 1705030200-30
2024/01/11 22:30:00.742840 [DVR] Running commercial detection on file 892 (TV/Burgers, Brew & 'Que/Burgers, Brew & 'Que S05E08 2019-03-19 OG Flavor 2024-01-11-2200.mpg)
2024/01/11 22:31:43.298585 [DVR] Commercial detection for Burgers, Brew & 'Que S05E08 2019-03-19 OG Flavor 2024-01-11-2200.mpg finished with 6 markers in 1m42.562165617s (12 threads).
2024/01/11 22:39:06.971432 [TNR] Cancelling stream 1313B64C/1 ch663 after no data was received for 6s
2024/01/11 22:39:06.987610 [TNR] Closed connection to 1313B64C/1 for ch663 Cooking Channel
2024/01/11 22:39:07.006511 [SNR] Signal statistics for "TV/Burgers, Brew & 'Que/Burgers, Brew & 'Que 2019-12-17 Champion Burger Knockouts 2024-01-11-2230.mpg": ss=100% snq=100% seq=100% bps=12426505,3809632-18434528 pps=1054,0-1578
2024/01/11 22:39:07.025481 [SNR] Buffer statistics for "TV/Burgers, Brew & 'Que/Burgers, Brew & 'Que 2019-12-17 Champion Burger Knockouts 2024-01-11-2230.mpg": buf=0% drop=0%
2024/01/11 22:39:07.140494 [DVR] Job 1705030200-30 Burgers, Brew & 'Que ended prematurely: 20m52.859524275s
2024/01/11 22:39:07.158114 [DVR] Starting job 1705030200-30 Burgers, Brew & 'Que on ch=[663 6109]
2024/01/11 22:39:07.158272 [DVR] Waiting 1h20m52.841728381s until next job 1705035600-28 American Restoration
2024/01/11 22:39:07.639991 [TNR] Opened connection to 1313B64C/0 for ch663 Cooking Channel
2024/01/11 22:39:07.642717 [DVR] Recording for job 1705030200-30 from 1313B64C ch663 into "TV/Burgers, Brew & 'Que/Burgers, Brew & 'Que 2019-12-17 Champion Burger Knockouts 2024-01-11-2230.mpg" for 20m52.841627039s
2024/01/11 23:00:00.005508 [TNR] Closed connection to 1313B64C/0 for ch663 Cooking Channel
2024/01/11 23:00:00.025115 [SNR] Signal statistics for "TV/Burgers, Brew & 'Que/Burgers, Brew & 'Que 2019-12-17 Champion Burger Knockouts 2024-01-11-2230.mpg": ss=100% snq=100% seq=100% bps=12502451,3226080-18100640 pps=1070,276-1550
2024/01/11 23:00:00.038759 [SNR] Buffer statistics for "TV/Burgers, Brew & 'Que/Burgers, Brew & 'Que 2019-12-17 Champion Burger Knockouts 2024-01-11-2230.mpg": buf=0% drop=0%
2024/01/11 23:00:00.183813 [DVR] Finished job 1705030200-30 Burgers, Brew & 'Que
2024/01/11 23:00:00.204169 [DVR] Waiting 59m59.795831815s until next job 1705035600-28 American Restoration
2024/01/11 23:00:00.233363 [DVR] Processing file-893: TV/Burgers, Brew & 'Que/Burgers, Brew & 'Que 2019-12-17 Champion Burger Knockouts 2024-01-11-2230.mpg
2024/01/11 23:00:00.617478 [IDX] Generating video index for file-893: TV/Burgers, Brew & 'Que/Burgers, Brew & 'Que 2019-12-17 Champion Burger Knockouts 2024-01-11-2230.mpg
2024/01/11 23:00:00.639321 [DVR] Running commercial detection on file 893 (TV/Burgers, Brew & 'Que/Burgers, Brew & 'Que 2019-12-17 Champion Burger Knockouts 2024-01-11-2230.mpg)
2024/01/11 23:00:25.041889 [IDX] Finished video index generation for file-893 in 24s
2024/01/11 23:01:39.983609 [DVR] Commercial detection for Burgers, Brew & 'Que 2019-12-17 Champion Burger Knockouts 2024-01-11-2230.mpg finished with 6 markers in 1m39.351545937s (12 threads).

20240112-02:59:59 Tuner: tuner1 tuning 663 Cooking Channel (auto:195MHz-713)
20240112-03:00:00 Tuner: tuner1 streaming http to 10.69.10.22:35562
20240112-03:00:00 CableCARD: tuner1 663 Cooking Channel (auto:195MHz-713) access = subscribed
20240112-03:29:59 Tuner: tuner1 http stream ended (remote closed)
20240112-03:29:59 Tuner: tuner1 tuning 663 Cooking Channel (auto:195MHz-713)
20240112-03:29:59 Tuner: tuner1 streaming http to 10.69.10.22:57894
20240112-03:30:00 CableCARD: tuner1 663 Cooking Channel (auto:195MHz-713) access = subscribed
20240112-03:39:06 Tuner: tuner1 http stream ended (remote closed)
20240112-03:39:06 Tuner: tuner0 tuning 663 Cooking Channel (auto:195MHz-713)
20240112-03:39:06 Tuner: tuner0 streaming http to 10.69.10.22:43924
20240112-03:39:06 CableCARD: tuner0 663 Cooking Channel (auto:195MHz-713) access = subscribed
20240112-03:59:59 Tuner: tuner0 http stream ended (remote closed)

I'm setting up live tv in Plex (running in a VM on the same server as channels) to see if Plex has the same issues...

[eric]

The internal buffer of the HDHomeRun is a fraction of a second, so if we haven't received any new data in 6 seconds, the video data during that time period was most definitely lost. If any other client/player does not display an error/notification of an issue like that, it just means they're masking it and silently ignoring this sort of problem.

We have aggressive timeouts because it's better to know about these sorts of failures than have them happen silently.

[Milky]

@eric

We have aggressive timeouts because it's better to know about these sorts of failures than have them happen silently.

I definitely agree with that logic, and honestly, If the buffer from the HDHR is that short, wouldn't Channels be better off running an even shorter timeout? It seems less of the recording would be lost this way. That is my biggest issue with all of this...

What is interesting is, while watching live via channels, When it does drop, I won't get any kind of pause or anything in video. It simply just says disconnected, press play to try again.

I recorded 12 hours of video with Plex yesterday, and also had Plex streaming from HDHR for about 8 hours straight on my phone (car auctions are great background noise). The signal never dropped on the phone. I never heard a pause in audio or anything noticeable. There are also no disconnect-reconnect entries in the HDHR Log. I know it is a small-ish sample size, but compared to channels, I'd be lucky to make it to 3 hours of either recording or streaming without a drop.

I know better than to assume Channels has to be problem here, but I'm trying to rule things out.

@eric

if we haven't received any new data in 6 seconds, the video data during that time period was most definitely lost.

May I ask, where in the chain would channels give that error? For example, it seems like Plex dumps the raw feed to .ts files on disk while recording (which takes up a metric crap-ton of space). From there, it will encode it when finished unless the encode while streaming option is checked. Then it will still dump the files to a temp folder while encoding at the same time.

I guess I am asking, is it possible there's something obscure about my setup that could be causing this beyond a simple loss of feed from the HDHR? Let's say Channels is having trouble reading from a disk or mem buffer AFTER the feed is received. Would that cause the same error? I could try installing channels on bare metal, but as I've said before, I have zero network or stability issues with many other way more demanding services, operating systems and containers running in virtualized environments. *(barring the occasional self-inflicted issues)

Thanks so much for trying to help with this. I FAR prefer Channels' interface over Plex for DVR and Live streaming content. When it works, it works really, really well.

[tmm1]

Can you stream reliably with tuner sharing off?

[Milky]

@tmm1

Can you stream reliably with tuner sharing off?

I turned it off and streamed over 15 hours over the past few days. Not a single disconnect.

I still had a few interrupted recordings though.

[tmm1]

Okay so the network path between the HDHR and playback device is fine.

You have some issue in the path between the HDHR and the DVR VM

[Milky]

Clearly the signal is making it to my 24p switch ok if other clients aren't disconnecting. From there to the server is literally a 1 meter, 10 gigabit sfp+ cable to an Intel server NIC.

It seems channels may be doing something different than anything else because I had the same symptoms in docker as well as having channels on its own vm.

I'm going to try changing the network adapter type for the Channels DVR Server virual machine in Proxmox to see if that changes anything. It is currently using VirtIO. I'll switch to Intel E1000 emulation to see if anything improves.

[Milky]

E1000 emulation didn't help.

In Proxmox, I set up multique for the network adapter of the VM, allocated more cores and ram ( originally 8 cores and 8gb of ram, now doubled. ) and knock-on-wood, I haven't had any interruptions since.

I will update again if need be.

[bnhf]

@Milky

In Proxmox, I set up multique for the network adapter of the VM, allocated more cores and ram ( originally 8 cores and 8gb of ram, now doubled. ) and knock-on-wood, I haven't had any interruptions since.

I use Proxmox as well, though my Channels DVR VM is Windows 11 based, with these resources:

• 8 cores (i9-13900 CPU)
• 32GB RAM
• NVMe PCIe Passthrough (Windows boot drive)
• VirtIO SCSI Controller (DrivePool data drives)
• VirtIO Network Controller (10Gbps SFP+ adapter)

Just giving you this a reference point, as this setup has been rock solid for me (running ~1yr with this config) -- with no multiqueue.

[Milky]

Out of all of the recordings over the past 30+ hours, I've had only one interruption. I would obviously prefer that to be zero, but progress is being made.

@bnhf

I use Proxmox as well, though my Channels DVR VM is Windows 11 based, with these resources:

Thanks for the reference.

I could easily spin up a windows 11 vm to see if that changes things...

Your CPU can handle hardware encoding, which I'm sure helps as mine cannot. The 2x Xeon E5-2683 v4 are a generation too old I believe.

Other than the ZFS cache, the server is only using about 160GB of the 384GB of ram, so I can allocate more to see if that helps.

I could also give it a few more cores too, but the VM's peak CPU usage over the past 24 hours is only 60%, which should leave plenty of headroom.

All my VMs are on mirror of 2TB Enterprise Samsung 983a NVME drives. The main storage is on TrueNAS in a ZFS pool with 4 16tb 7200 rpm WD Red Pros (32tb mirrored vdev). That pool has 4x Intel optane SSDs for caching, as well as intel SSDs for Special Small Blocks.

Peak overall server CPU usage is only 30%, and a peak I/O delay of 0.34% over the past 24 hours. The server overall is definitely not stressed.

Based on this discussion and troubleshooting, it seems my issue is probably somewhere in the network. The server NIC has 2x 10G SFP+ as well as 2x 1G regular ol' RJ45 Ethernet. I could pass through one of the Intel I350 Gigabit nics directly to the VM. It might help to rule out something going on with the proxmox virtualization of the hardware, but I'm doubtful that is the issue as I have so many other things running with this setup without issue.

[Milky]

Just an update:

It seems everything I changed or tweaked has worked. I've gone an entire week recording HOURS of content with out a single interruption.

I'm adding a second HDHR Prime now that everything is working as expected.

Thanks for all the time, and for the great product.