Need Help - ChannelsDVR Won't Start After Upgrade

Hi,

I can't get Channels DVR to start on my Pi (v3). Had been working fine for months, and stopped today. Looking at the logs, it looks to have started an upgrade from v2021.08.12.0054 to v2021.10.25.1801

I can't seem to get the service to start. Any tips/tricks? How can I rollback to the previous version?

Status just stalls on "activating"

pi@raspberrypi:~ $ systemctl status channels-dvr
● channels-dvr.service - Channels DVR
Loaded: loaded (/etc/systemd/system/channels-dvr.service; enabled; vendor preset: enabled)
Active: activating (auto-restart) (Result: signal) since Mon 2021-10-25 19:32:56 CDT; 9s ago
Process: 2619 ExecStart=/bin/sh -c exec /home/pi/channels-dvr/latest/channels-dvr >> /home/pi/channels-dvr/data/channels-dv
Main PID: 2619 (code=killed, signal=SEGV)
lines 1-5/5 (END)

What does these show:

tail -50 ~/channels-dvr/data/channels-dvr.log
uname -a

pi@raspberrypi:~ $ tail -50 ~/channels-dvr/data/channels-dvr.log
2021/10/25 09:48:00.357212 [DVR] Fetching guide data for 43 stations in USA-OTA73170 @ 2021-11-09 2:00PM
2021/10/25 09:48:03.339704 [DVR] indexed 348 airings (43 channels) [0s fetch, 2s index]
2021/10/25 09:48:03.775125 [DVR] indexed 7 movies (3 channels) [0s fetch, 0s index]
2021/10/25 09:48:03.791299 [DVR] Fetching guide data for 8 stations in USA-DITV-DEFAULT @ 2021-11-09 2:00PM
2021/10/25 09:48:04.361177 [DVR] indexed 36 airings (8 channels) [0s fetch, 0s index]
2021/10/25 09:48:04.387916 [DVR] Fetching guide data for 159 stations in X-TVE @ 2021-11-09 2:00PM
2021/10/25 09:48:11.815160 [DVR] indexed 1087 airings (159 channels) [0s fetch, 6s index]
2021/10/25 09:48:12.695340 [DVR] indexed 51 movies (22 channels) [0s fetch, 0s index]
2021/10/25 09:48:12.727201 [DVR] Fetching guide data for 43 stations in USA-OTA73170 @ 2021-11-09 8:00PM
2021/10/25 09:48:15.265652 [DVR] indexed 329 airings (43 channels) [0s fetch, 1s index]
2021/10/25 09:48:15.970969 [DVR] indexed 10 movies (3 channels) [0s fetch, 0s index]
2021/10/25 09:48:15.986868 [DVR] Fetching guide data for 8 stations in USA-DITV-DEFAULT @ 2021-11-09 8:00PM
2021/10/25 09:48:16.826465 [DVR] indexed 42 airings (8 channels) [0s fetch, 0s index]
2021/10/25 09:48:16.903039 [DVR] Fetching guide data for 159 stations in X-TVE @ 2021-11-09 8:00PM
2021/10/25 09:48:25.212603 [DVR] indexed 1096 airings (159 channels) [1s fetch, 7s index]
2021/10/25 09:48:26.218908 [DVR] indexed 46 movies (18 channels) [0s fetch, 0s index]
2021/10/25 09:48:26.249686 [DVR] Fetching guide data for 43 stations in USA-OTA73170 @ 2021-11-10 2:00AM
2021/10/25 09:48:28.891271 [DVR] indexed 388 airings (43 channels) [0s fetch, 2s index]
2021/10/25 09:48:29.563582 [DVR] indexed 2 movies (1 channels) [0s fetch, 0s index]
2021/10/25 09:48:29.579171 [DVR] Fetching guide data for 8 stations in USA-DITV-DEFAULT @ 2021-11-10 2:00AM
2021/10/25 09:48:30.364932 [DVR] indexed 43 airings (8 channels) [0s fetch, 0s index]
2021/10/25 09:48:30.389104 [DVR] Fetching guide data for 159 stations in X-TVE @ 2021-11-10 2:00AM
2021/10/25 09:48:39.206019 [DVR] indexed 1243 airings (159 channels) [1s fetch, 7s index]
2021/10/25 09:48:40.267759 [DVR] indexed 32 movies (14 channels) [0s fetch, 0s index]
2021/10/25 09:48:55.972678 [IDX] Pruned 47 expired groups from USA-OTA73170 in 181.157277ms.
2021/10/25 09:48:56.173432 [IDX] Pruned 7 expired groups from USA-DITV-DEFAULT in 200.604587ms.
2021/10/25 09:48:56.480273 [IDX] Pruned 140 expired groups from X-TVE in 302.971846ms.
2021/10/25 09:48:57.255438 [IDX] Pruned 1306 expired airings from USA-OTA73170 in 458.6371ms.
2021/10/25 09:48:57.530950 [IDX] Pruned 124 expired airings from USA-DITV-DEFAULT in 275.309936ms.
2021/10/25 09:48:59.281113 [DVR] Waiting 10h40m30.718950092s until next job 1635211770-16 American Dad
2021/10/25 09:48:59.798583 [IDX] Pruned 4345 expired airings from X-TVE in 2.267432745s.
2021/10/25 10:23:10.209202 [M3U] stream timestamps: FS1: start_at=2021-10-25T10:22:27-05:00 current_at=2021-10-25T10:22:35-05:00 end_at=2021-10-25T10:22:43-05:00
2021/10/25 10:23:10.214045 [TNR] Opened connection to M3U-GMAStream for ch103 FS1
2021/10/25 10:23:14.702539 [TNR] Closed connection to TVE-DTV for ch6074 FBN
2021/10/25 10:25:33.264176 [TNR] Closed connection to M3U-GMAStream for ch103 FS1
2021/10/25 10:25:33.365338 [M3U] stream timestamps: ESPN: start_at=2021-10-25T10:24:50-05:00 current_at=2021-10-25T10:25:04-05:00 end_at=2021-10-25T10:25:09-05:00
2021/10/25 10:25:33.365487 [TNR] Opened connection to M3U-GMAStream for ch100 ESPN
2021/10/25 11:38:09.306630 [TNR] Closed connection to M3U-GMAStream for ch100 ESPN
2021/10/25 11:38:12.542378 [TVE] stream timestamps: fbn: start_at=2021-10-25T11:37:12-05:00 current_at=2021-10-25T11:37:32-05:00 end_at=2021-10-25T11:37:41-05:00
2021/10/25 11:38:12.542847 [TNR] Opened connection to TVE-DTV for ch6074 FBN
2021/10/25 11:39:28.198383 [TNR] Closed connection to TVE-DTV for ch6074 FBN
2021/10/25 12:00:08.574525 [TVE] stream timestamps: fbn: start_at=2021-10-25T11:59:07-05:00 current_at=2021-10-25T11:59:34-05:00 end_at=2021-10-25T11:59:38-05:00
2021/10/25 12:00:08.576241 [TNR] Opened connection to TVE-DTV for ch6074 FBN
2021/10/25 12:09:34.238141 [ERR] ULK invalid: {"next_time": -1, "error": "sess not found"}
2021/10/25 13:44:34.302414 [TNR] Closed connection to TVE-DTV for ch6074 FBN
2021/10/25 15:47:00.273048 [SYS] Downloading new version v2021.10.25.1801
2021/10/25 15:47:18.020086 [SYS] Update downloaded and verified successfully.
2021/10/25 15:47:18.020185 [SYS] Shutting down for upgrade from v2021.08.12.0054 to v2021.10.25.1801
2021/10/25 15:47:18.021467 [SYS] Bonjour service stopped.
2021/10/25 15:47:18.046044 [DVR] Recording engine stopped.

pi@raspberrypi:~ $ uname -a
Linux raspberrypi 4.19.75-v7+ #1270 SMP Tue Sep 24 18:45:11 BST 2019 armv7l GNU/Linux

Thanks. It appears the new build is failing to boot like you said.

Please run these and post the output:

~/channels-dvr/latest/channels-dvr
dmesg | tail -10

pi@raspberrypi:~ $ ~/channels-dvr/latest/channels-dvr
Segmentation fault
pi@raspberrypi:~ $ dmesg | tail -10
[ 1317.200741] Under-voltage detected! (0x000d0005)
[ 1321.360682] Voltage normalised (0x00080000)
[ 1587.600791] Under-voltage detected! (0x00050005)
[ 1591.760795] Voltage normalised (0x00000000)
[ 1693.681341] Under-voltage detected! (0x00050005)
[ 1697.841332] Voltage normalised (0x00000000)
[ 2367.604404] Under-voltage detected! (0x00050005)
[ 2371.764378] Voltage normalised (0x00000000)
[ 2729.526227] Under-voltage detected! (0x00050005)
[ 2733.686177] Voltage normalised (0x00000000)
pi@raspberrypi:~ $

Gross. We will need to do some debugging to figure this out. Have you used gdb before?

apt install gdb
gdb ~/channels-dvr/latest/channels-dvr

run
bt
quit

To get back up and running, do this:

cd ~/channels-dvr
ln -nsf 2021.08.12.0054 latest
chmod 444 2021.10.25.1801/*
systemctl restart channels-dvr

Nope, but happy to give it a shot. Starting now, will post update once finished

Thank you for your help, I am working on the old version.

Is there a way I can disable the automatic update to prevent this from occurring again?

The instructions I provided will disable the auto update. (chmod sets to readonly which should cause the updater to fail).

I would like to figure out how to fix the new build however, since we will not keep supporting the old version.

Thank you!

I am happy to supply any logs or data you need, just let me know.

Were you able to run any of those gdb commands? That would show us where the crash is happening.

Also what version of raspbian are you running?

Well crap. Yes, I ran them. But didn't copy and paste it out...

Any harm in running those gdb commands again?

Do it this way now:

chmod 755 ~/channels-dvr/2021.10.25.1801/channels-dvr
gdb ~/channels-dvr/2021.10.25.1801/channels-dvr

In the gdb prompt you basically need to run, wait for the crash, then bt will show where it crashed. Copy paste all that and you can quit out.

Buster

PRETTY_NAME="Raspbian GNU/Linux 10 (buster)"
NAME="Raspbian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=raspbian
ID_LIKE=debian

pi@raspberrypi:~ $ gdb ~/channels-dvr/2021.10.25.1801/channels-dvr
GNU gdb (Raspbian 8.2.1-2) 8.2.1
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "arm-linux-gnueabihf".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/.
Find the GDB manual and other documentation resources online at:
http://www.gnu.org/software/gdb/documentation/.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/pi/channels-dvr/2021.10.25.1801/channels-dvr...(no debugging symbols found)...done.
(gdb) run
Starting program: /home/pi/channels-dvr/2021.10.25.1801/channels-dvr
[New LWP 3986]
[New LWP 3987]
[New LWP 3988]
[New LWP 3989]
[New LWP 3990]
[New LWP 3991]
2021/10/25 20:22:26 couldn't open settings.db: timeout
[LWP 3990 exited]
[LWP 3991 exited]
[LWP 3988 exited]
[LWP 3987 exited]
[LWP 3986 exited]
[LWP 3983 exited]
[Inferior 1 (process 3983) exited with code 01]
(gdb) bt
No stack.
(gdb)

Now to restart channels again, do I just run:

systemctl restart channels-dvr

Or do I need to chmod that directory back to read only first

What steps should I take to get it back to running again? Seems to just be stalling out again on startup