Multiple Recordings Interrupted

Channels DVR Server Troubleshooting
#1

I have had an issue where multiple recordings show as interrupted and i have corruption when watching them. This happened on sun 26th and Sunday 19th.

This is when my Channels DVR hosted on a Synology DS1815+ is recording multiple programs, playing back programs and doing commercial detection. My source is a hd home run cable tuner.

After the issue on the 19th i scaled back commercial detection from 3 to 2 threads as CPU was a little high. That didn't fix it (i scaled it back to 1 thread last night to see if that helps).

The logs on the cable card tuner say that DVR disconnected nothing about an inherent problem with the cable stream.

The logs on the channels DVR imply the stream was interrupted. I see tons of these in the logs for sunday:

2021/10/24 22:01:22.901482 [TNR] Cancelling stream 13222164/1 ch513 after no data was received for 6s
2021/10/24 22:01:22.914525 [TNR] Closed connection to 13222164/1 for ch513 KCPQ FOX HD
2021/10/24 22:01:23.159302 [SNR] Statistics for "TV/Family Guy/Family Guy S20E05 Brief Encounter 2021-10-24-2127.mpg": ss=100% snq=98%,91%-100% seq=100% bps=10281532,7053760-12872736 pps=604,0-1102 neterr=12%
2021/10/24 22:01:23.159767 [DVR] Job 1635136020-1 Family Guy ended prematurely: 1m36.840242918s
2021/10/24 22:01:23.424211 [DVR] Starting job 1635136020-1 Family Guy on ch=[513]
2021/10/24 22:01:23.424282 [DVR] Waiting 18h55m36.575725493s until next job 1635206220-15 NFL Football
2021/10/24 22:01:24.153036 [TNR] Opened connection to 13222164/1 for ch513 KCPQ FOX HD
2021/10/24 22:01:24.153145 [DVR] Recording for job 1635136020-1 from 13222164 ch513 into "TV/Family Guy/Family Guy S20E05 Brief Encounter 2021-10-24-2127.mpg" for 1m36.575647394s
2021/10/24 22:01:48.358193 [TNR] Cancelling stream 13222164/1 ch513 after no data was received for 6s
2021/10/24 22:01:48.371177 [TNR] Closed connection to 13222164/1 for ch513 KCPQ FOX HD
2021/10/24 22:01:48.655908 [SNR] Statistics for "TV/Family Guy/Family Guy S20E05 Brief Encounter 2021-10-24-2127.mpg": ss=100% snq=100% seq=100% bps=8286162,1502496-10353536 pps=575,0-887 neterr=8%
2021/10/24 22:01:48.656336 [DVR] Job 1635136020-1 Family Guy ended prematurely: 1m11.343683s
2021/10/24 22:01:48.910737 [DVR] Starting job 1635136020-1 Family Guy on ch=[513]
2021/10/24 22:01:48.910805 [DVR] Waiting 18h55m11.089202612s until next job 1635206220-15 NFL Football
2021/10/24 22:01:49.406039 [TNR] Opened connection to 13222164/1 for ch513 KCPQ FOX HD
2021/10/24 22:01:49.406143 [DVR] Recording for job 1635136020-1 from 13222164 ch513 into "TV/Family Guy/Family Guy S20E05 Brief Encounter 2021-10-24-2127.mpg" for 1m11.089141803s
2021/10/24 22:02:11.819984 [TNR] Cancelling stream 13222164/1 ch513 after no data was received for 6s
2021/10/24 22:02:11.833157 [TNR] Closed connection to 13222164/1 for ch513 KCPQ FOX HD
2021/10/24 22:02:12.107169 [SNR] Statistics for "TV/Family Guy/Family Guy S20E05 Brief Encounter 2021-10-24-2127.mpg": ss=100% snq=100% seq=90%,0%-100% bps=7754350,0-8924736 pps=541,0-765 neterr=9%
2021/10/24 22:02:12.107627 [DVR] Job 1635136020-1 Family Guy ended prematurely: 47.892382322s
2021/10/24 22:02:12.281887 [DVR] Starting job 1635136020-1 Family Guy on ch=[513]
2021/10/24 22:02:12.281953 [DVR] Waiting 18h54m47.71805401s until next job 1635206220-15 NFL Football
2021/10/24 22:02:12.906011 [TNR] Opened connection to 13222164/1 for ch513 KCPQ FOX HD
2021/10/24 22:02:12.906118 [DVR] Recording for job 1635136020-1 from 13222164 ch513 into "TV/Family Guy/Family Guy S20E05 Brief Encounter 2021-10-24-2127.mpg" for 47.717992751s
(tbh after seeing these in the logs i am amazed the stream was somewhat watchable...)

Last night I tried recording several things at once and all was OK.

I am not aware of any underlying issue with networking so continue to operate on the thesis thet Synology was 'stressed'; but both memory and CPU seemed ok on sunday.

Any other ideas what i should look for?

1 Like
#2

Is it only that single cable channel that has the issue, or more than one channel?
If more than one channel and you say the HDHR system logs don't show anything unusual (timestamps there are UTC time) it has to be the network path or the Synology.

#3

thanks, i record two channels simultaneously last night, no issues
(yeah i figured out the UTC timestamps, lol)

but both sundays i recorded simpsons (fox) + familly guy (fox) + equalizer (CBS) + watched an NFL game that was also recording at same time and it was wonky on both CBS and FOX.

on the 19th when i did it it was doing commercial detection too (so the NFL game was recording, but when it stopped simpsons it started commercial detection, and so on).

This week i didn't know i had an issue until monday night so i couldn't poke around and see what was happening.

I may try and repro the sunday case of recording on two channels while also watching others and doing commercial detection....

1 Like
#4

here is a comskip.ini you can try

thread_count=2
play_nice_sleep=5
play_nice_start=1959
play_nice_end=2259
verbose=1
lowres=2
logo_fraction=0.39
output_vdr=1
output_edl=1
output_ffmeta=1
output_ffsplit=1
; See comskip.txt in the distribution zip file for many settable parameters, read manual.html, tuning.html and debugwindow.html for how to tune and debug comskip
detect_method=111	; 1=black frame, 2=logo, 4=scene change, 8=fuzzy logic, 16=closed captions, 32=aspect ration, 64=silence, 128=cutscenes, 255=all
validate_silence=1	; Default, set to 0 to force using this clues if selected above.
validate_uniform=1	; Default, set to 0 to force using this clues (like pure white frames) if blackframe is selected above.
validate_scenechange=1	; Default, set to 0 to force using this clues if selected above.
verbose=1	; Show a lot of extra info, level 5 is also OK, set to 0 to disable
max_brightness=60 ; Frame not black if any pixels checked are greater than this (scale 0 to 255)
test_brightness=40 ; Frame not pure black if any pixels checked are greater than this, will check average brightness (scale 0 to 255)
max_avg_brightness=25	; Maximum average brightness for a dim frame to be considered black (scale 0 to 255) 0 means autosetting
maxbright=1
brightness_jump=200
max_commercialbreak=600 ; Maximum length in seconds to consider a segment a commercial break
min_commercialbreak=24	; Minimum length in seconds to consider a segment a commercial break
max_commercial_size=197	; Maximum time in seconds for a single commercial or multiple commercials if no breaks in between
min_commercial_size=4 ; Mimimum time in seconds for a single commercial
min_show_segment_length=250 ; Any segment longer than this will be scored towards show.
non_uniformity=500	; Set to 0 to disable cutpoints based on uniform frames
max_volume=500	; Any frame with sound volume larger than this will not be regarded as black frame
max_silence=100	; 
min_silence=12	; Any deep silence longer than this amount of frames is a possible cutpoint
noise_level=5
ticker_tape=0	; Amount of pixels from bottom to ignore in all processing 
logo_at_bottom=0	; Set to 1 to search only for logo at the lower half of the video, do not combine with subtitle setting
ignore_side=0
subtitles=0
use_existing_logo_file=1 ; Use logo file already created at beginning of show, default 0
two_pass_logo=1
punish=1	; Compare to average for sum of 1=brightness, 2=uniform 4=volume, 8=silence, 16=schange, set to 0 to disable
punish_threshold=1.3	; Multiply when amount is above average * punish_threshold
punish_modifier=4	; When above average * threshold multiply score by this value
intelligent_brightness=0 ; Set to 1 to use a USA specific algorithm to tune some of the settings, not adviced outside the USA
logo_percentile=0.92	; If more then this amount of logo is found then logo detection will be disabled
logo_threshold=0.775	; Digital signals can be higher
global_threshold=1.79
min_black_frames_for_break=1	; 
punish_no_logo=1	; Default, set to 0 to avoid show segments without logo to be scored towards commercial
aggressive_logo_rejection=0
connect_blocks_with_logo=1	; Set to 1 if you want successive blocks with logo on the transition to be regarded as connected, set to 0 to disable
delay_logo_search=0	; 
logo_filter=0 ; Set the size of the filter to apply to bad logo detection, 4 seems to be a good value.
edge_step=2	; New, default 1
cut_on_ar_change=1	; Set to 1 if you want to cut also on aspect ratio changes when logo is present, set to 2 to force cuts on aspect ratio changes. set to 0 to disable
delete_show_after_last_commercial=0	; Set to 1 if you want to delete the last block if its a show and after a commercial
delete_show_before_first_commercial=0	; New, default 0
delete_show_before_or_after_current=0	; Set to 1 if you want to delete the previous and the next show in the recording, this can lead to the deletion of trailers of next show
delete_block_after_commercial=0	; Set to max size of block in seconds to be discarded, set to 0 to disable 
remove_before=0	; Amount of seconds of show to be removed before ALL commercials
remove_after=0	; Amount of seconds of show to be removed after ALL commercials
shrink_logo=5	; Reduce the duration of the logo with this amount of seconds
after_logo=0	; Set to number of seconds after logo disappears comskip should start to search for silence to insert an additional cutpoint
before_logo=0
padding=0.50
ms_audio_delay=5
volume_slip=20
skip_b_frames=1	; Set to 1 to force Comskip to skip frames for higher processing speed.
hardware_decode=1	; Set to 1 to enable hardware accelerated video decoding, only available in donator version
max_repair_size=200	; Will repair maximum 200 missing MPEG frames in the timeline, set to 0 to disable repairing for players that don't use PTS. 
disable_heuristics=4	; Bit pattern for disabling heuristics, adding 1 disables heristics 1, adding 2 disables heristics 2, adding 4 disables heristics 3, 255 disables all heuristics 
delete_logo_file=1	; Set to 1 if you want comskip to tidy up after finishing
output_framearray=0	; Create a big excel file for detailed analysis, set to 0 to disable
output_data=0	; Create a dump of the user data channel, used for CC and XDS (such as V-Chip info). Can be use together with output_framearray to remote debug CC decoding
output_videoredo=0	; The old videoredo format
output_videoredo3=0	; The new videoredo v3 format.
output_womble=0
output_mls=0	; Set to 1 if you want MPeg Video Wizard bookmark file output
output_cuttermaran=0
output_mpeg2schnitt=0
output_mpgtx=0
output_dvrcut=0
output_zoomplayer_chapter=0
output_zoomplayer_cutlist=0
output_dvrmstb=0	; Set to 1 if you're running DVRMS-Toolbox
output_edlx=0
output_vcf=0
output_bsplayer=0
output_btv=0	; Set to 1 if you want Beyond TV chapter cutlist output
output_projectx=0	; Set to 1 if you want ProjectX cutlist output (Xcl)
output_avisynth=0
output_demux=0	; Set to 1 if you want comskip to demux the mpeg file while scanning
sage_framenumber_bug=0
sage_minute_bug=0
#5

@djcastaldo thanks Dave, what does this comskip.ini change from default? (i have never looked at comskip settings in detail even though been using it since the days of MCE, lol)

#6

a few things. It does a playnice to slow down comskip from 1959 to 2259. Outside of these hours, it also uses a few lowres settings for faster detection, but tbh I haven't used the default comskip for many years and have only been using a few slightly tweaked versions of what I posted above.

Anyway, just save the above to /volume1/@appstore/ChannelsDVR/channels-dvr/data/comskip.ini
If you don't like it, just delete it and Channels will go back to using its built-in default.

1 Like
#7

thanks, done, do i need to restart the DVR service?

#8

no. it should just use that .ini as long as it is there in the /data dir.

#9

Bad network connection between the DVR and HDHR

1 Like
#10

Thanks, i have tried moving the hdhomerun to a different switch port on a different device - will see if that fixes it (it was directly on my UDM-PRO router, which i found in a funky state last night, so i moved it to my main switch).

What is neterr measuring?

1 Like
#11

There's a dropped packets counter that's available via the hdhomerun's stats over UDP. Its the hdhomerun saying it had to drop packets because they could not be delivered fast enough.

1 Like
#12

Thanks.

Moving hd-homerun to my core switch didn’t help. When recording on two or three channels and maybe when commercial detection kicks in I am seeing neterr issues. Pinging the HDHomeRun during this time at 0.2 of a second intervals reveals no packet drops from a WiFi client.

I am stumped. Will try repro creation again and this time ping from the Synology to the hd home run. I saw issues when cpu on Synology was running at 75%. Not sure where the bottle kneck or network issue is :frowning:

#13

In the second test tonight I had 3 recordings going, one commercial detection and one playing of a recorded item. No issues at all.

As such I think this confirms it’s a network or Synology resource issue I am yet to track down. Thanks for your help all. I will keep an eye on it and report back if I find it.

1 Like
#14

hi, i just moved to a new NAS that supports transcoding (and channels dvr lit up the transcode UI for streaming).

I note in the comskip.ini you provided it seems to have hardware_decode=1 - can i assume that means the comskip process now uses that?

BTW love you comskip.ini, seems to have removed most of my mis-detects too, thanks!

#15

yes, thats what it does.

1 Like
#16

just to close off

i did the following and one or more of these fixed the issue:

  • moved DVR to a NAS that was one switch away from the homeruns switch (it was 3 switches away)
  • moved from synology DS1815+ to DS918+
  • enabled HW transcoding of commercials - the network error always occurred when commercial detection was happening, despite CPU on DS1815+ never being pegged.
  • updated my ubiquiti unifi software / firmware's to latest pre-release version

at this point i will call it as solved as I haven't seen the issue in 3 days.

3 Likes