Pi OS Upgrade Disaster

sawc · November 7, 2021, 2:28pm

I also emailed this to the support email, but wasn't sure where would be the best or most responsive place, so I apologize for double messages.

I am using the official Pi image from Channels and booting from a USB drive, I have been running it for a little over a year.

Last night I saw there was an update for the OS portion of my Pi DVR so I applied it, I hadn't seen one since April. It rebooted but never came back. There was a brief time I saw the UI via the web and it showed the new version but it was waiting on NTP sync. After that all I see is connection refused messages in the browser.

I then moved the Pi to a place with a monitor so I could see it boot, it seemed to be stuck at a message about "wait until kernel time synchronized"

Last night I started to backup the drive and recordings to my PC so I could just do a fresh install and restore, but it errored out at some point so this morning I started to see if I could get the DVR working again via SSH. I must be doing something wrong or it isn't loading the public keys correctly. I either get an error about denied due to public key, or I get connection refused like I see in the browser.

I went back to the direct connection to a monitor, and this latest attempt to boot I see it seems to have loaded farther, but still nothing working. It says to contact support, so that is what I am doing. Hopefully we can rescue the install as the recording transfer will take quite some time.

Sorry if this is kind of all over the place, I just wanted to make sure I shared all the things I had been trying and saw. I am a server administrator so I do understand a good amount of tech, but Linux and Pi are for sure not my strong areas.

tmm1 · November 7, 2021, 5:52pm

Do you have a keyboard attached?

Can you hit enter then type: systemctl status

sawc · November 7, 2021, 5:56pm

I was able to get SSH working, generated new keys using OpenSSH client instead of puttygen.

[root@dvr-server ~]$ systemctl status
● dvr-server
    State: degraded
     Jobs: 0 queued
   Failed: 1 units
    Since: Wed 1969-12-31 18:00:02 CST; 51 years 10 months ago
   CGroup: /
           ├─init.scope
           │ └─1 /sbin/init
           └─system.slice
             ├─rngd.service
             │ └─328 /usr/sbin/rngd -f
             ├─systemd-udevd.service
             │ └─143 /usr/lib/systemd/systemd-udevd
             ├─rauc.service
             │ └─326 /usr/bin/rauc --mount=/run/rauc service
             ├─wpa_supplicant.service
             │ └─333 /usr/sbin/wpa_supplicant -u
             ├─systemd-journald.service
             │ └─124 /usr/lib/systemd/systemd-journald
             ├─winbind.service
             │ ├─438 /usr/sbin/winbindd --foreground --no-process-group
             │ └─494 /usr/sbin/winbindd --foreground --no-process-group
             ├─NetworkManager.service
             │ └─322 /usr/sbin/NetworkManager --no-daemon
             ├─system-ha\x2dcli.slice
             │ └─[email protected]
             │   ├─ 355 /bin/bash /usr/sbin/hassos-cli
             │   ├─4837 n/a
             │   └─4838 [hassos-cli]
             ├─wsdd2.service
             │ └─334 /usr/sbin/wsdd2 -w -b vendor:Channels,model:DVR server,vendorurl:getchannels.com,modelurl:getchannels.com/dvr-server
             ├─systemd-resolved.service
             │ └─316 /usr/lib/systemd/systemd-resolved
             ├─tailscaled.service
             │ └─374 /usr/sbin/tailscaled --state=/var/lib/tailscale/tailscaled.state --socket=/run/tailscale/tailscaled.sock --port 41641
             ├─dbus.service
             │ └─118 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
             ├─systemd-timesyncd.service
             │ └─318 /usr/lib/systemd/systemd-timesyncd
             ├─dropbear.service
             │ ├─ 360 /usr/sbin/dropbear -R -E -p 22222 -s
             │ ├─1927 /usr/sbin/dropbear -R -E -p 22222 -s
             │ ├─1928 -bash
             │ ├─4823 systemctl status
             │ └─4828 less
             └─systemd-logind.service
               └─330 /usr/lib/systemd/systemd-logind

tmm1 · November 7, 2021, 5:58pm

systemctl status channels-dvr

ls -alh /mnt/data/channels-dvr

sawc · November 7, 2021, 6:00pm

[root@dvr-server ~]$ systemctl status channels-dvr
● channels-dvr.service - Channels DVR
     Loaded: loaded (/usr/lib/systemd/system/channels-dvr.service; enabled; vendor preset: enabled)
     Active: inactive (dead)

Feb 02 09:31:18 dvr-server systemd[1]: Dependency failed for Channels DVR.
Feb 02 09:31:18 dvr-server systemd[1]: channels-dvr.service: Job channels-dvr.service/start failed with result 'dependency'.

[root@dvr-server ~]$ ls -alh /mnt/data/channels-dvr
drwxr-xr-x    3 root     root          27 Nov  1 16:11 .
drwxr-xr-x    3 root     root          35 Nov  1 16:11 ..
drwxr-xr-x    2 channels channels       3 Nov  1 16:11 data

tmm1 · November 7, 2021, 6:06pm

df -h

distro-upgrade --dvr 2021.11.06.2348

sawc · November 7, 2021, 6:07pm

[root@dvr-server ~]$ df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/root               157.4M    157.4M         0 100% /
devtmpfs                  1.8G         0      1.8G   0% /dev
tmpfs                     1.8G         0      1.8G   0% /dev/shm
tmpfs                   748.1M     16.6M    731.5M   2% /run
tmpfs                     4.0M         0      4.0M   0% /sys/fs/cgroup
tmpfs                   748.1M     16.6M    731.5M   2% /etc/machine-id
tmpfs                     1.8G         0      1.8G   0% /media
tmpfs                     1.8G         0      1.8G   0% /shares
/dev/zram1               31.0M    764.0K     28.0M   3% /var
/dev/zram2               15.5M     80.0K     14.3M   1% /tmp
/dev/sda1                31.9M      7.8M     24.1M  24% /mnt/boot
/dev/sda7                89.0M      3.9M     78.4M   5% /mnt/overlay
/dev/sda7                89.0M      3.9M     78.4M   5% /etc/docker
/dev/sda7                89.0M      3.9M     78.4M   5% /etc/dropbear
/dev/sda7                89.0M      3.9M     78.4M   5% /etc/modprobe.d
/dev/sda7                89.0M      3.9M     78.4M   5% /etc/modules-load.d
/dev/sda7                89.0M      3.9M     78.4M   5% /etc/udev/rules.d
/dev/sda7                89.0M      3.9M     78.4M   5% /root/.docker
/dev/sda7                89.0M      3.9M     78.4M   5% /root/.ssh
/dev/sda7                89.0M      3.9M     78.4M   5% /var/lib/bluetooth
/dev/sda7                89.0M      3.9M     78.4M   5% /var/lib/samba
/dev/sda7                89.0M      3.9M     78.4M   5% /var/lib/systemd
/dev/sda7                89.0M      3.9M     78.4M   5% /var/lib/tailscale
/dev/sda7                89.0M      3.9M     78.4M   5% /etc/NetworkManager/system-connections
/dev/sda7                89.0M      3.9M     78.4M   5% /etc/hostname
/dev/sda7                89.0M      3.9M     78.4M   5% /etc/hosts
/dev/sda7                89.0M      3.9M     78.4M   5% /etc/systemd/timesyncd.conf
/dev/sda7                89.0M      3.9M     78.4M   5% /etc/systemd/system.control
/dev/sda7                89.0M      3.9M     78.4M   5% /etc/timezone
/dev/sda7                89.0M      3.9M     78.4M   5% /root/.bash_history
/dev/sda7                89.0M      3.9M     78.4M   5% /var/lib/NetworkManager
/dev/root               157.4M    157.4M         0 100% /usr/share/zoneinfo/posix/Etc/UTC

[root@dvr-server ~]$ distro-upgrade --dvr 2021.11.06.2348
Downloading Channels DVR 2021.11.06.2348 (linux-arm64) to /mnt/data/channels-dvrmkdir: can't create directory 'channels-dvr/2021.11.06.2348': Read-only file system

ERROR: Installation of Channels DVR failed.
ERROR: Please contact [email protected] with the full output above.

tmm1 · November 7, 2021, 6:10pm

Seems like /mnt/data is not mounted for some reason.

systemctl start mnt-data.mount
systemctl status mnt-data.mount

sawc · November 7, 2021, 6:15pm

The first command sat for a while, it didn't seem to finish. I did a CTRL + C and then put the 2nd command.

[root@dvr-server ~]$ systemctl start mnt-data.mount

systemctl status mnt-data.mount
^C
[root@dvr-server ~]$ systemctl start mnt-data.mount
^C
[root@dvr-server ~]$ systemctl status mnt-data.mount
● mnt-data.mount - HassOS data partition
     Loaded: loaded (/usr/lib/systemd/system/mnt-data.mount; enabled; vendor preset: enabled)
     Active: inactive (dead)
      Where: /mnt/data
       What: /dev/disk/by-label/hassos-data

Feb 02 09:31:18 dvr-server systemd[1]: Dependency failed for HassOS data partition.
Feb 02 09:31:18 dvr-server systemd[1]: mnt-data.mount: Job mnt-data.mount/start failed with result 'dependency'.
Nov 07 11:54:55 dvr-server systemd[1]: Dependency failed for HassOS data partition.
Nov 07 11:54:55 dvr-server systemd[1]: mnt-data.mount: Job mnt-data.mount/start failed with result 'dependency'.
Nov 07 12:12:28 dvr-server systemd[1]: Dependency failed for HassOS data partition.
Nov 07 12:12:28 dvr-server systemd[1]: mnt-data.mount: Job mnt-data.mount/start failed with result 'dependency'.

If you feel that just reimaging and then copying my database and TV folder backup to do a restore is what I am going to end up needing to do, I can do that. I am also fine with a bit of troubleshooting if this seems salvageable in a respectable amount of time for both of us.

sawc · November 7, 2021, 6:54pm

My family really wanted this back up and running so I just went with the reimage to get things up as quick as possible. I am in the process now.

I appreciate your responsiveness as always, you and the other staff are top notch!

Hopefully the next time I need to do an OS upgrade it will go as expected

sawc · November 7, 2021, 7:22pm

I noticed that I see in the TV show section all the shows I had recorded, however to speed transfer times up I only copied back some of them. How can I cleanup the list so it only shows the ones that actually still exist?