Recovering from a Catastrophic Failure

Thought I'd share the experiences and terrors of my past week+ recovering from a complete and catastrophic failure of my server and backup external HDD within days of each other! Let my exhaustion and frustration be beautiful learning lessons for you, or at least schadenfreude if that is your thing.

The Wonderful and Easy
Installing a new Channel DVR on another machine and restoring from backup is as easy as could be. I had done some test runs in the past just to see, and was glad it worked just as well for real. I already cloud-sync'd my Imports directory, so every Stream Link was saved. I was able to get my old server to boot up (took 2 days) long enough to get the Database, Images, and Streaming directories off of there, which I now also cloud-sync after learning I should have been doing that the hard way. In case you are wondering how to do this, you can create a "Symbolic Link" where one directory is actually in two places. I do this with the above mentioned Channels directory to a folder on my Onedrive so I can interact with it on there on any of my other computers. Here are directions for Windows:

I don't have enough free cloud storage to do this with the Logs directory, so that I'm just going to be backing up to an external SSD. While my server was up, I did manually move that over, as well as do all of the other functions of migration like turning off Bonjour on the old server.

The Fairly Painless
While my External HDD I used for backups and overflows decided to die as well just a few days later, I had a second smaller External SSD with much of my non-Channels related backups on it (Fitness programs, music, photos, etc...). I was able to replicate the directory structure on that for what I used with Channels. For anything that was personal media, most of the time I could re-download it and place it in the same location (made the drive the same letter as the old one). At the worst Channels thought it was new and I just had to run a prune to delete the old version.

For Channels recordings, if I was able to... acquire... another copy of the file, all I had to do was get the file name from Options > View Details:

Then I named my replacement file the exact same thing, put it in the appropriate Channels recording directory, and refreshed the metadata, which you can do en masse per season. Afterwards, it was fine:

This worked even if my new file was a different extension (I just gave it the expected extension). All the files play with no issue. There is a problem of alignment with detected commercials and things like that, but that is a minor inconvenience and can just be ignored.

Also, some recordings could just be replaced with Stream Links, so I just deleted the original recordings and went with those where it made sense.

I have a new larger SSD coming in the mail and I'll be moving all of my backups and overflows on to that and maintaining that second backup as a NAS for harder to find things, my personal stuff, and non-Channels related files.

The Annoying but Oh Well
Some shows had not been watched yet and I could not find replacements for them. For some of those, the easiest thing to do was to modify the pass and chose to re-record deleted episodes (in some cases, changing Advanced Options to only record after a particular season if the show had been started). I deleted the episodes through Settings > DVR Database > Manage Recordings Database > Missing/Deleted. It is VERY IMPORTANT to note that this just moves the records to the Trash, it does not mark them as not deleted.

Also, there is no picking and choosing (its all or nothing), so I had to restore individual items I was not quite done with yet.

I do have to remember that once everything is recorded again that I will have to go back in and change the options to not re-recorded deleted episodes, otherwise we'll be in a loop forever.

If something were more individual of a recording and it was in the guide, it was very, very easy to mark as not recorded and then select to record again. However, it was per episode/movie and was dependent upon the guide. When I could do it, it was the best. Most of the time, this feature did nothing for me.

Finally, some recordings/files I had were rare and/or I wanted to save them forever. Others, I have been carrying for decades and still haven't watched. This gave me a chance to confront my own digital hoarding and let some things go.

The Downright Horrible
As hinted at above, not everything was going to work out this way. So that brings us to the CURL -XDELETE method of marking things as not recorded (special thanks to @chDVRuser for the many, many helpful posts during my recovery process):

This is plain awful. Having to find an individual program ID and delete it one-by-one from the recording database was by the far the hardest and most painful thing to do, especially in the volumes I was dealing with. I cannot emphasize enough how much this feature is needed in the front end:

And as it is, I have things that this still may not work for. For instance, I have some movies I recorded that are not airing any time soon and are hard to come by (like 1930s stuff). If I could just say re-record this and be done, it would be fine. I am attempting an Advanced Pass but I have no way right now to confirm it is working.

image

Instead, I will have to keep an eye out to see if some of these movies are airing, and if so that my pass will work.

Further, really advanced things like what is available under /dvr and what type of curl or other commands work are just scattered over so many different threads on here that it is insanely difficult to find out if a function is already possible. I know this is not something the Devs would want in the general help, but somewhere else where it could be documented and organized would be wonderful. I'm not as super advanced as some of the users on here, but enough that I can follow what they say. Maybe a wiki would be a place for something like this?

The Still to Come
My new server is on my much more powerful daily driver. I have certainly noticed performance improvements in a lot of ways, especially in Comm Skip detection. I suppose I'm also happy to have one less always-on PC around here. This has inspired me to get another PC that I am basically using as a screen for a TV off of the network and combine its functions into my daily driver, as well. So less electricity use and fewer running things will be nice.

Unfortunately, my office is nowhere near where my router needs to be, and even WiFi 6 on the 5ghz spectrum has too much jitter and latency for things the size of OTA recordings (though live hasn't been that much of a problem). My MOCA adapters are in the mail, too, so hopefully that will resolve all of these issues. In the meantime, I've changed my preference orders to use Local TVE over OTA to limit the moving pieces and reduce sizes, so that is also something else I'll have to flip back.

And as mentioned above, I still have to deal with the joy of moving 800gb of data from one SSD to another and remapping their drive letters so as far as Channels (and Plex) are concerned everything is where it was before and nothing changed.

So that's it! Some lessons, some stories, some suggestions, and some finality to come...

4 Likes

Sorry. That really sucks. Glad you didn't have to start completely over. I've been there before and it was really painful. Now I'm off to review my disaster recovery plan :wink:

2 Likes

Sorry to hear about your experience, but thanks for the comprehensive explanation.

I can't understand why the devs haven't commented on this. It just makes sense to put the button where it's most useful and needed. Do they feel otherwise? Is it too hard? Is it on their 2do list?

The devs have mentioned that the end-points/api's are for their internal use, undocumented and could change at any time. I keep those I run across in text files on my PC for future reference (along with the url of the posts).

I wouldn't want to start a topic on them as it would be a pain to keep up. Especially if they changed something without notification, which is expected since they're undocumented.

Given that this a non-standard procedure, I don’t know why the devs should comment. If I depart from established norms I understand that I’m on my own when it comes to backup and recovery.

You're missing the point.
Have you read my post he refers to?

Perhaps I overreacted. Sounded like complaining because your unsupported workaround, albeit cool and useful, made disaster recovery difficult. That’s the kind of thing that comes with unsupported workarounds - especially when they’re really cool. But reading it again, it seems like advocating to make the workaround official or make it easier which is fair enough. I just wonder if this situation illustrates the many things the devs have to consider when rolling out a new feature.

1 Like

And I wouldn't want a single person to have to keep that up, either, which is why I think a user-maintained Wiki would be more doable.

In this case, I remembered that you had posted things about how to delete individual recording records, but I didn't remember what or where since I never had an occasion to use that, so I began searching through your posts until I found one. But I didn't just want to launch into doing it without testing that CURL worked in a more benign way, so I started searching through posts about CURL commands until I found one that did a basic retrieval of a search through the guide. Once I confirmed that worked and I had the language as expected, then I was more willing to do the dangerous stuff.

Now imagine there was a Wiki maintained by the users on here instead and there was just a page called "CURL commands" with examples like that. Or I just read this post from @Jay1987:

So instead of trying to manage in a post, people could put it all in an MCE Buddy page. The same thing with managing the Pluto Docker. I always return to the post about how to disable the Docker, do the upgrade, and re-enable. Or @racameron gave me an idea of creating multiple passes of the same program in different priorities to get the recording potentials in the order I wanted it. This is a potential feature, but an advanced use case that could be documented with examples and referred to in the future instead of having to re-post the same idea.

So things like these would be useful to be slightly more organized and less discussion based, but at the same time not "officially" supported by the Devs and not dependent upon one person to keep up.

1 Like

There are so many benign uses for "Mark as not recorded and record again". A sports event can go long, a news preemption may happen, a power outage happens in the middle--the list goes on and on. Disaster recovery is certainly a more extreme example, but there are plenty of day-to-day reasons one would need to re-record.

Maybe from the Dev's perspective they see it as there is a way to re-record because they gave us both "Mark as not recorded" and the "Record Again" buttons:

image

But both are guide-dependent, and that is the real problem. What we are looking for here is this function, but at the recording level itself. More of a "set it and forget it" so we don't have to watch the guide for the opportunity to do so.

2 Likes

First of all, it's not an unsupported workaround.
The functionality already exists as a button, but the only place you can find it is on that same episode in the 16 day guide. If it airs again in the next 16 days. You have to find it in the guide to get to the button.

I'm advocating for them to add the button to the recorded episode in DVR > Manage and Library.
Obviously when you have the need for that button, you are looking at the recorded episode and have determined there's something wrong with it (there are many reasons) and you want the dvr to forget it recorded it so your pass can record it again when it airs.

1 Like

No argument here. Your idea doesn't address an issue that am currently dealing with but that doesn't matter. I just think this is flirting with being off topic for this thread. But who am I to criticize? I hijack threads all the time :grin:

The MoCa network is amazing (amazing what you can do with jitter drops from 150 to 1.5), the SSD drive directly connected to my new "server" is supremely faster, and everything is running great! I decided some of my backups only need to live on my NAS and don't need to be on my "Media Drive", so that has also made a lot more space for me. I've been re-recording my missing episodes without much issue and have maybe a few dozen things left to still track down or catch a recording of. So overall, we are back in business!

Ha - here I am wondering if "Catastrauphic" is German. Carry on! :slight_smile:

Damn it, spell checker, why don't you work in the title?

1 Like

Well, I learned that the Logs\recording folder (of which mine was 10 gigs) is only important in troubleshooting and can safely have its contents be deleted. As such, I created a script that runs daily at 3am to delete anything older than 7 days. Now the entire directory is much smaller and reasonable, so I created the same symbolic link from above and it is backing up to my OneDrive.

1 Like