Pi DVR: read-only filesystem

@tmm1 It happened again. Everything was fine, and then while copying some of the many things that got deleted back to the Imports folder, abruptly SMB stopped working, and then recordings and everything else stopped too with read only file system.

I rebooted and it fixed itself briefly, then did it again.

Logs have been submitted as e26ab83e-f838-43c0-a040-b642ff0cae3c

Plugged the drive into a Mac which automatically ran fsck_exfat and mounted normally, then rebooted. All looks fine again except:

Randomly about 1TB is missing.

Weekend project may be starting over and copying all of these files back from a clean volume.

From your diagnostics:

Feb 25 11:29:21 dvr-server systemd-fsck[209]: exfatprogs version : 1.0.4
Feb 25 11:29:21 dvr-server systemd-fsck[209]: ERROR: /Imports/Movies: cluster is already allocated for the other file. truncated to 0 bytes. Truncate (y/N)? n
Feb 25 11:29:21 dvr-server systemd-fsck[209]: ERROR: /Database/backup-20210224.154321/recorder.db: cluster is already allocated for the other file. truncated to 786432 bytes. Truncate (y/N)? n
Feb 25 11:29:21 dvr-server systemd-fsck[209]: /dev/sda9: checking stopped. directories 1117, files 4675
Feb 25 11:29:21 dvr-server systemd-fsck[209]: /dev/sda9: files corrupted 2, files fixed 0
Feb 25 11:29:21 dvr-server systemd-fsck[203]: fsck failed with exit status 4.
Feb 25 11:32:56 dvr-server kernel: exFAT-fs (sda9): error, found bogus dentry(110) beyond unused empty group(108) (start_clu : 144, cur_clu : 144)
Feb 25 11:32:56 dvr-server kernel: exFAT-fs (sda9): Filesystem has been set read-only

You submitted logs after the reboot?

Were you using SMB from a Mac to transfer files?

I submitted logs after the reboot, but before I plugged it into the Mac.

Yes, I was using SMB from a Mac to transfer files.

Submitted logs now, from after the (second) reboot, after plugging it into the Mac.

Logs have been submitted as f8fecdee-97d5-4292-85e9-e506e0324223 .

I'd really like to catch the logs before rebooting if possible, so we can see what the original trigger is. The kernel messages seem to get lost on reboot.

I wonder if something is going on with Samba. Were you using that a lot before too whenever it seemed to crap out?

I was using Samba for maybe 30 minutes to copy a folder over. I took the opportunity while the drive was connected to the Mac to just copy everything over directly to the drive which is much, much faster.

I would say that in general now with the new exfat driver, everything is VERY SLOW. SMB was very slow, opening the client is very slow, even the web interface can be very slow to start, and comskip is glacial.

Huh...

It's the same driver, just with some bug fixes included.

I suspect it just may be that it was corrupt (including the missing 1 TB). I copied everything off last night and am copying everything back. Of course the ExFAT driver on the Mac is also slow and this is taking hours. Will keep you posted.

1 Like

So, that took forever. For some reason channels kept crashing when I tried to restore the database. I deleted the most recent one and still had a problem (this happened on MacOS too though it just ran a processor at 98% for over an hour with no end in sight.) I removed everything except the second most recent one and that restored.

After many mishaps, also I had success with rsync.

Something is still not quite right on the SMB implementation (there are files that show up in ls over ssh that don't show up with SMB), but I'll live with it for now. Happy to be back online, and also have a backup in case this abruptly happens again!

Thanks,
ajp

@adamandaj any more problems since you rebuilt last time?

No, thanks! Everything is great though I am still obsessively rsyncing just in case :slight_smile:
Thank you!

I have had this happen twice now - I am current unable to record or play anything

I have not set up ssh access - is there any remedy for this issue?

After two installs lost the ability to write to the disk I have abandoned my attempt to use the PI4 as my DVR.

I am now using my Synology DS1819 NAS with no issues what so ever.

Updating the operating system by long-pressing on Check for Updates under Operating System updates the ExFAT drivers and for me, appears to have solved the problem.

I have setup now 6x Pi with the Channels DVR image, 2x I use personally.

Never had this issue, but have read about similar issue related to using external storage via usb in general.

I suspect it has to do with whatever usb drive and or enclosure you were using.
Either it was not computable fully with the Pi, or needs more power, meaning you have to connected a powered usb 3.0 hub first, then the drive. (there is a website that lists all the discovered bad enclosures that the pis do not like)

I use a Samsung T7 ssd on one pi, and a Samsung 850evo ssd in a clear enclosure on the other, both flashed to the Channels Image and both work great with no issues with the file systems access. The only prep i did on the drives was use the erase feature in the Pi Imager software, the 850evo was used, had things on it already, the T7 was new out of box.

I did almost immediately update the OS to the latest version, but i did test things first, the samba share etc, at first install.

New, and I think related problem. Everything is slow-- really slow. There used to be no problem recording two or three TVE streams while watching something else, and now more than one activity at a time just grinds everything to a halt. Recording more than two TVE streams leads constantly to interrupted and corrupted recordings. Playback while something is recording takes 20-30 seconds to start and then stutters unwatchably. Skipping a commercial is another 20-30 seconds.

The web interface gets stuck.

When I do an rsync to back up generally speeds are 20 mb a second or more, as high as 40, but then abruptly will drop to 500k or less for awhile.

Not the drive -- fsck is fine and I don't think it's mechanically failing.

Any suggestions? I was going to wipe, reformat and restore but the copying alone will take a day so I am inclined to avoid that!

Thanks

Sounds pretty bad. One thing we could try is a kernel upgrade to something more modern. I can put together a test build tomorrow.

There are some reports on other forums about this being related to power. Just to rule this out, I'm going to try a powered hub tonight (it's the recommended drive and didn't need it for the past four months-- this just started-- but can't hurt to look into it.)

1 Like