DVR is constantly failing to clean up streaming cache. I have the Streaming directory symlinked to use SSD storage for quick scrubbing with tuner sharing, the read/write permissions are correct. This only happens on Unraid, same setup on Freenas works fine 99% of the time.
2022/02/10 08:40:51.943294 [ERR] Failed to clean up /DVR/Streaming/ch1108-dANY-78c6b5c22e1a-1264691194: unlinkat /DVR/Streaming/ch1108-dANY-78c6b5c22e1a-1264691194/cache: directory not empty
It's probably something about the way that Go's filesystem libraries are handling the link traversal. You might want to give a bind mount a try instead.
(And for those finding this thread for similar issues, just remember that unsupported setups are prone to unexpected situations. YMMV.)
This is a known issue. We try to clean up the directories when a stream completes but have run into some race conditions with ffmpeg finishing.
I'll try to circle back to fix this or at least clean it up after a period of time has passed to prevent the empty (or almost empty) directories from sticking around.
If it's racing with ffmpeg and the clean up is a background task that isn't time sensitive then how about putting in a wait state of a chunk of time (5 min?) before trying to clean out directories?
Or go nuclear and do the equivalent of rm -rf 
Funny enough, rm -rf is not guaranteed to succeed when files are being written at the same time as the delete is happening.
This issue is still something on our list to improve.
That is odd. I'd think that any process writing to the file would have a file handle open and would continue to write even after the file was unlinked. That's how we do our logging of sensitive data. Create a tempfile, open a file handle to it, delete the file. Since we have a handle we can read and write to it until our process ends.
But we don't have to deal with various file systems so things are a bit easier.
This has always been a problem. I raised the issue multiple times in the past. I just ended up using this simple one line script below to run as a cronjob every morning at 4am to clean it up. I actually forgot about it until I just read this post.
#!/bin/bash
find /mnt/user/ChannelsDVR/Streaming -mindepth 1 ! -regex '^/mnt/user/ChannelsDVR/Streaming/m3u8(/.*)?' -delete
(it chopped it up to two lines. Basically add a space after regex before ' but it should all be on one line.)
The race condition is slightly different.
The way rm -rf or any other thing removes a directory and everything in it is basically:
readdir() to get all of the children of a directoryunlink() each of the childrenunlink() the directoryThe race condition is when a separate thing goes and writes a file in the directory after (1) has happened but before (3) has happened. It causes the unlink() of the directory to fail.
I believe there is something going on with our cleanup of the ffmpeg processes that sometimes leads to this behavior, but have also seen errors where there are special files starting with .nfs that we are unable to delete. We are trying to identify what the issues are with our general cleanup process, but also intend to implement something similar to what @Marino13 described to clean up anything left over.
Ugh. Yeah that's a problem. I didn't draw the connection that the unlink failed because the directory was truly not empty. I don't think there is any way to keep the kernel from erroring with 'directory not empty'.
Race condition between parent and a child seems ..... unavoidable?