Channels DVR v2019.12.05.0128 consistent crash immediately after startup

Channels DVR Server Troubleshooting
#1

Hello--

I've been using Channels DVR for quite a while and have never experienced something like this. Every time, immediately after starting, Channels DVR will crash. Here's the log:

2020/01/24 12:34:07 [SYS] Starting Channels DVR v2019.12.05.0128 (freebsd-x86_64 pid:69331) in /usr/local/channels-dvr/data

2020/01/24 12:34:08 [HDR] Found 1 devices

2020/01/24 12:34:14 [TLS] Requesting certificate for 849e010702a3.channelsdvr.net

2020/01/24 12:34:14 [SYS] Started HTTP Server

2020/01/24 12:34:14 [DVR] Recording engine started in /media/recordings

2020/01/24 12:34:14 [SYS] Bonjour service running for dvr-channels-dvr.local. [192.168.0.15]

2020/01/24 12:34:14 [NAT] Successfully mapped port 8089 using natpmp

2020/01/24 12:34:15 [SYS] Created database snapshot: backup-20200124.123414

panic: runtime error: slice bounds out of range [:2242793858906825446] with capacity 22158611

goroutine 57 [running]:

github.com/blevesearch/bleve/index/scorch/segment/zap.(*Segment).readMM(0xc000ca7180, 0x1f200121c091a2dc, 0x1f200121c091a2e6, 0x0, 0xc000f96db0, 0xc0003403a8)

github.com/blevesearch/[email protected]/index/scorch/segment/zap/mem.go:64 +0x1de

github.com/blevesearch/bleve/index/scorch/segment/zap.(*SegmentBase).readMem(...)

github.com/blevesearch/[email protected]/index/scorch/segment/zap/mem.go:53

github.com/blevesearch/bleve/index/scorch/segment/zap.(*PostingsList).read(0xc000b438c0, 0x1f200121c091a2dc, 0xc0013e8ba0, 0xc000b438c0, 0xc0014b92f0)

github.com/blevesearch/[email protected]/index/scorch/segment/zap/posting.go:292 +0x3fa

github.com/blevesearch/bleve/index/scorch/segment/zap.(*Dictionary).postingsListFromOffset(0xc0013e8ba0, 0x1f200121c091a2dc, 0x0, 0xc000b438c0, 0x0, 0xc000f2a000, 0x1247)

github.com/blevesearch/[email protected]/index/scorch/segment/zap/dict.go:72 +0x6a

github.com/blevesearch/bleve/index/scorch/segment/zap.persistMergedRest(0xc0008b5900, 0xa, 0xa, 0xc0008b58b0, 0xa, 0xa, 0xc000b04fc0, 0x1c, 0x1c, 0xc0005ff5c0, ...)

github.com/blevesearch/[email protected]/index/scorch/segment/zap/merge.go:315 +0x12e6

github.com/blevesearch/bleve/index/scorch/segment/zap.MergeToWriter(0xc0008b5900, 0xa, 0xa, 0xc0008b58b0, 0xa, 0xa, 0x400, 0xc0005ff590, 0xc0008faa80, 0x203000, ...)

github.com/blevesearch/[email protected]/index/scorch/segment/zap/merge.go:140 +0x31f

github.com/blevesearch/bleve/index/scorch/segment/zap.MergeSegmentBases(0xc0008b5900, 0xa, 0xa, 0xc0008b58b0, 0xa, 0xa, 0xc0008f8390, 0x2c, 0x400, 0xc0008faa80, ...)

github.com/blevesearch/[email protected]/index/scorch/segment/zap/merge.go:81 +0x24c

github.com/blevesearch/bleve/index/scorch/segment/zap.Merge(0xc0008b5860, 0xa, 0xa, 0xc0008b58b0, 0xa, 0xa, 0xc0008f8390, 0x2c, 0x400, 0xc0008faa80, ...)

github.com/blevesearch/[email protected]/index/scorch/segment/zap/merge.go:56 +0x121

github.com/blevesearch/bleve/index/scorch.(*Scorch).planMergeAtSnapshot(0xc0006b4c00, 0xc00043f300, 0xc0008b5810, 0x0, 0x0)

github.com/blevesearch/[email protected]/index/scorch/merge.go:206 +0x95d

github.com/blevesearch/bleve/index/scorch.(*Scorch).mergerLoop(0xc0006b4c00)

github.com/blevesearch/[email protected]/index/scorch/merge.go:61 +0x247

created by github.com/blevesearch/bleve/index/scorch.(*Scorch).Open

github.com/blevesearch/[email protected]/index/scorch/scorch.go:172 +0x125

I can't update via the GUI because it crashes too quickly to be usable.

Im running as an iocage plugin on Freenas 11.2-U7.

Can anyone suggest steps to fix?

Thanks again
PP

#2

Gross. I think your guide db has gotten corrupted. Delete the USA* folders in the iocage jail.

#3

Thank you! I deleted 2 dirs beginning with USA and restarted the jail...and Channels-DVR appears to be working and usable once again.

Thank you SO MUCH for your quick response and help...much apprecated!

have a great night
PP

1 Like
#4

Out of curiosity, how did you determine that the guide DB was corrupted? Would love to know based upon the logs I supplied...

thanks!
PP

#5

Back again....

It appears the app is crashing again, this time with a different failure mode. Is this another issue related to a corrupted guide db?

Here's the log

2020/01/24 19:30:50 [SYS] Starting Channels DVR v2019.12.05.0128 (freebsd-x86_64 pid:83870) in /usr/local/channels-dvr/data

2020/01/24 19:30:50 [HDR] Found 1 devices

2020/01/24 19:30:56 [TLS] Requesting certificate for 849e010702a3.channelsdvr.net

2020/01/24 19:30:56 [SYS] Started HTTP Server

2020/01/24 19:30:56 [DVR] Recording engine started in /media/recordings

2020/01/24 19:30:56 [SYS] Bonjour service running for dvr-channels-dvr.local. [192.168.0.15]

2020/01/24 19:30:56 [NAT] Successfully mapped port 8089 using natpmp

2020/01/24 19:30:56 [SYS] Created database snapshot: backup-20200124.193056

panic: runtime error: index out of range [8] with length 8

goroutine 53 [running]:

github.com/blevesearch/bleve/index/scorch/segment.(*MemUvarintReader).ReadUvarint(0xc0008e6780, 0x2, 0x0, 0x0)

github.com/blevesearch/[email protected]/index/scorch/segment/int.go:130 +0xd6

github.com/blevesearch/bleve/index/scorch/segment/zap.(*PostingsIterator).readFreqNormHasLocs(0xc000c36000, 0x0, 0x14c, 0xbb7101, 0x0, 0x0)

github.com/blevesearch/[email protected]/index/scorch/segment/zap/posting.go:427 +0x107

github.com/blevesearch/bleve/index/scorch/segment/zap.(*PostingsIterator).nextAtOrAfter(0xc000c36000, 0x0, 0xc000ef1178, 0x2, 0x2, 0x0)

github.com/blevesearch/[email protected]/index/scorch/segment/zap/posting.go:546 +0xf1

github.com/blevesearch/bleve/index/scorch/segment/zap.(*PostingsIterator).Next(...)

github.com/blevesearch/[email protected]/index/scorch/segment/zap/posting.go:519

github.com/blevesearch/bleve/index/scorch/segment/zap.mergeTermFreqNormLocs(0xc000b4e2a0, 0xc00052c980, 0xa, 0x10, 0xc000c36000, 0xc000b6e000, 0x1f6, 0x1f6, 0xc000ef15d0, 0xc000ef16b8, ...)

github.com/blevesearch/[email protected]/index/scorch/segment/zap/merge.go:515 +0x59d

github.com/blevesearch/bleve/index/scorch/segment/zap.persistMergedRest(0xc000b4b480, 0x7, 0x7, 0xc000b4b400, 0x7, 0x7, 0xc000a04000, 0x1c, 0x1c, 0xc000b4e2a0, ...)

github.com/blevesearch/[email protected]/index/scorch/segment/zap/merge.go:329 +0x15bc

github.com/blevesearch/bleve/index/scorch/segment/zap.MergeToWriter(0xc000b4b480, 0x7, 0x7, 0xc000b4b400, 0x7, 0x7, 0x400, 0xc000b4e030, 0xc000042540, 0x203000, ...)

github.com/blevesearch/[email protected]/index/scorch/segment/zap/merge.go:140 +0x31f

github.com/blevesearch/bleve/index/scorch/segment/zap.MergeSegmentBases(0xc000b4b480, 0x7, 0x7, 0xc000b4b400, 0x7, 0x7, 0xc000c52000, 0x2c, 0x400, 0xc000042540, ...)

github.com/blevesearch/[email protected]/index/scorch/segment/zap/merge.go:81 +0x24c

github.com/blevesearch/bleve/index/scorch/segment/zap.Merge(0xc000b4b3c0, 0x7, 0x7, 0xc000b4b400, 0x7, 0x7, 0xc000c52000, 0x2c, 0x400, 0xc000042540, ...)

github.com/blevesearch/[email protected]/index/scorch/segment/zap/merge.go:56 +0x121

github.com/blevesearch/bleve/index/scorch.(*Scorch).planMergeAtSnapshot(0xc000be8000, 0xc000184f80, 0xc0007fe320, 0x0, 0x0)

github.com/blevesearch/[email protected]/index/scorch/merge.go:206 +0x95d

github.com/blevesearch/bleve/index/scorch.(*Scorch).mergerLoop(0xc000be8000)

github.com/blevesearch/[email protected]/index/scorch/merge.go:61 +0x247

created by github.com/blevesearch/bleve/index/scorch.(*Scorch).Open

github.com/blevesearch/[email protected]/index/scorch/scorch.go:172 +0x125

#6

Just a WAG, but looking at the errors, the crash seems related to Bleve, which is a text search library. So, it makes sense that this additional issue is still related to the guide DB ...

#7

Thanks for the reply....

So I cleared the data/USA* directories...and then I started clearing out stuff in usr/local/channels-dvr and it looks like I accidentally hosed myself. If I recall correctly, there is a symbolic link named latest which points to a backup directory named with a timestamp (like this: 2020.01.23.someOtherNumber). In other words, right now my usr/local/channels-dvr dir only contains data/ and install.sh and the application won't start at all now.

I executed ./insall.sh but that didn't seem to help

I can't recall to save my life how I installed channels-dvr in the Freenas iocage...is there an install guide someone can point me to?

Or would it be easier to get this application working again by deleting some other file?

Thx
PP

#8

If it helps this is mine. It should have directories for each release you have updated to with latest being a symbolic link to the version you want to use normally the latest but could be any of them.

root@channels-dvr:/ # ls -al /usr/local/channels-dvr/
total 158
drwxr-xr-x   9 dvr   dvr      11 Jan 23 01:05 .
drwxr-xr-x  19 root  wheel    19 Nov  1 15:56 ..
drwxr-xr-x   2 dvr   dvr       8 Jan 15 07:07 2020.01.15.0443
drwxr-xr-x   2 dvr   dvr       8 Jan 15 19:44 2020.01.15.2157
drwxr-xr-x   2 dvr   dvr       8 Jan 16 06:45 2020.01.16.0535
drwxr-xr-x   2 dvr   dvr       8 Jan 17 07:39 2020.01.17.0543
drwxr-xr-x   2 dvr   dvr       8 Jan 19 17:56 2020.01.19.2104
drwxr-xr-x   2 dvr   dvr       8 Jan 23 00:30 2020.01.23.0005
drwxr-xr-x  10 dvr   dvr      48 Jan 24 23:33 data
-rwxr-xr-x   1 root  dvr    1201 Nov  1 11:23 install.sh
lrwxr-xr-x   1 dvr   dvr      15 Jan 23 00:30 latest -> 2020.01.23.0005
root@channels-dvr:/ #

Searching FreeNAS on the forum should give you several results for setting up a new jail.

#9

Thanks for the response--much appreciated. I woke up and remembered I took a couple snapshots of the jail so just rolled back.

Per the instructions above about a corrupted guide, I rm -r the USA dirs before starting the iocage and am tailing the channels-dvr log. Seeing lots of guide database errors...is this normal? Here's a snip from the logs:

2020/01/25 07:41:31 [DVR] Fetching guide data for 309 stations in USA-CO05419-X @ 2020-02-04 1:00AM
2020/01/25 07:41:49 [DVR] indexed 3927 airings (309 channels) [1s fetch, 16s index]
2020/01/25 07:41:51 [DVR] indexed 66 movies (25 channels) [0s fetch, 2s index]
2020/01/25 07:41:52 [DVR] Fetching guide data for 309 stations in USA-CO05419-X @ 2020-02-04 7:00AM
2020/01/25 07:41:59 [ERR] Guide database failure: merging err: merging failed: invalid address -2965620380511248199/172373
2020/01/25 07:42:08 [DVR] indexed 3290 airings (309 channels) [1s fetch, 15s index]
2020/01/25 07:42:11 [DVR] indexed 127 movies (38 channels) [0s fetch, 1s index]
2020/01/25 07:42:11 [DVR] Fetching guide data for 309 stations in USA-CO05419-X @ 2020-02-04 1:00PM
2020/01/25 07:42:16 [ERR] Guide database failure: merging err: merging failed: invalid address -2965620380511248199/172373
2020/01/25 07:42:26 [DVR] indexed 3304 airings (309 channels) [1s fetch, 13s index]
2020/01/25 07:42:28 [ERR] Guide database failure: merging err: merging failed: invalid address -2965620380511248199/172373
2020/01/25 07:42:28 [DVR] indexed 175 movies (50 channels) [0s fetch, 1s index]
2020/01/25 07:42:29 [DVR] Fetching guide data for 309 stations in USA-CO05419-X @ 2020-02-04 7:00PM
2020/01/25 07:42:35 [ERR] Guide database failure: merging err: merging failed: invalid address -2965620380511248199/172373
2020/01/25 07:42:47 [ERR] Guide database failure: merging err: merging failed: invalid address -2965620380511248199/172373
2020/01/25 07:42:47 [DVR] indexed 3300 airings (309 channels) [1s fetch, 17s index]
2020/01/25 07:42:49 [DVR] indexed 171 movies (47 channels) [0s fetch, 1s index]

#10

Here are a bunch more messages of a slightly different nature

2020/01/25 07:49:05 [ERR] Guide database failure: merging err: merging failed: invalid address -2965620380511248199/172373

2020/01/25 07:49:10 [ERR] Guide database failure: merging err: merging failed: invalid address -2965620380511248199/172373

2020/01/25 07:49:16 [ERR] Guide database failure: merging err: merging failed: invalid address -2965620380511248199/172373

2020/01/25 07:49:21 [ERR] Guide database failure: merging err: merging failed: invalid address -2965620380511248199/172373

2020/01/25 07:49:26 [ERR] Guide database failure: merging err: merging failed: invalid address -2965620380511248199/172373

2020/01/25 07:49:31 [ERR] Guide database failure: merging err: merging failed: invalid address -2965620380511248199/172373

2020/01/25 07:49:36 [ERR] Guide database failure: merging err: merging failed: invalid address -2965620380511248199/172373

2020/01/25 07:49:41 [ERR] Guide database failure: merging err: merging failed: invalid address -2965620380511248199/172373

2020/01/25 07:49:46 [ERR] Guide database failure: merging err: merging failed: invalid address -2965620380511248199/172373

2020/01/25 07:49:51 [ERR] Guide database failure: merging err: merging failed: invalid address -2965620380511248199/172373

2020/01/25 07:49:56 [ERR] Guide database failure: merging err: merging failed: invalid address -2965620380511248199/172373

2020/01/25 07:50:01 [ERR] Guide database failure: merging err: merging failed: invalid address -2965620380511248199/172373

2020/01/25 07:50:06 [ERR] Guide database failure: merging err: merging failed: invalid address -2965620380511248199/172373

2020/01/25 07:50:11 [ERR] Guide database failure: merging err: merging failed: invalid address -2965620380511248199/172373

2020/01/25 07:50:16 [ERR] Guide database failure: merging err: merging failed: invalid address -2965620380511248199/172373

2020/01/25 07:50:21 [ERR] Guide database failure: merging err: merging failed: invalid address -2965620380511248199/172373

2020/01/25 07:50:27 [ERR] Guide database failure: merging err: merging failed: invalid address -2965620380511248199/172373

#11

Delete the USA* directories only.

If the errors start again it means you have some bad RAM or your disk is failing.

#12

Lol, it appears you're right! I just discovered a couple uncorrectable sectors in my pool.

Thanks again
PP

2 Likes