Power Outage/Server Crash

Am wondering what people have for backup options to their Channels DVR setup. I'm at my house in Italy and I normally watch US TV from my US house where I have Channels running on a QNAP NAS. We had a power outage yesterday at the US house and the QNAP NAS hasn't restarted so I am SOL both on Channels & Tailscale.

I managed to connect into my router and setup a OPENVPN connection between my Android TV stick in Italy and my US (ASUS) router and I bought a month on Sling TV so I am effectively back to "normal". Will see if I can resuscitate the QNAP when I get back or if I need to replace it. Either way, it's got me looking at weak points in my setup and what I can do to provide redundancy/backup.

1 Like

I'm in Portugal, and do the same via my home in the U.S.

I've spent significant effort on this issue, which has multiple aspects to it. A UPS is great, but only when part of a solution that includes orderly shutdown and startup.

Ultimately, I leveraged several existing open source projects, and created my own "orchestration" setup using Docker-based versions of control software for APC UPS units (of which I have 6).

Look at bnhf/apcupsd on Docker Hub and apcupsd-admin-plus on GitHub. The house is in a rural area with frequent power problems and this setup has saved my a** many, many times.

https://hub.docker.com/repository/docker/bnhf/apcupsd

Thanks - will look at that. My QNAP NAS was on a UPS but not setup for shutdown (it mainly covers for quick power glitches). Losing Channels is a problem but switching to a service (Sling or YTTV) is a decent stopgap. The bigger challenge was losing my VPN (tailscale) exit node. I also have an apartment in Manhattan so I'll probably put a backup tailscale exit node there so I get redundancy that way.

I wish the UPS guys would provide 12/18V DC output rather than AC. I have some small DC UPS for routers and cable modem and they work well. I think if APC or Cyberpower made a larger DC UPS they would sell well.

Agreed -- not having a residential exit node in the USA, is the bigger hassle. But you're on the right track with having another one in a different location. I do the same with a family residence in Minneapolis -- just-in-case.

If you're technically inclined, you can't go wrong setting up a Proxmox host, as you can use that for all kinds of things including having a Proxmox LXC setup as an exit node. You can spin-up VMs, LXCs or Docker containers from anywhere in the world, without having the limitations of using a VPS at data center. Their IPs are typically blocked, for anything entertainment or banking related.

Keep in mind when setting up an exit node, that whatever you use shouldn't be anything too low-end to handle Tailscale's encryption requirements. For example, having a router as an exit node will typically be a bottleneck. A RaspberryPi4 or better would be the minimum, and a decent x64 platform running Proxmox even better. :slight_smile:

I have a couple of extra old Lenovo tiny PCs (older i5's, 8/16GB RAM, 256GB SSD, WIN11) sitting around. One acts as a server at my Italy house. I'm toying with making one a Linux box as Tailscale runs in kernel mode on that, Probably use DietPi. That would be for the Italy house as I use it as a gateway when we're in the US so my wife can watch RAI (Italian TV).

The QNAP was fine and reasonably solid for that last 5-6 years and I was running it headless and with Tailscale (running in a container) it was no different managing it locally vs. remotely. I am curious to see why it didn't reboot. If it was a hardware failure, that's life but if for some reason it didn't boot when power was restored I want to see if it will start if I am there in person. The QNAPs can be a little finicky when not shutdown properly.

For Channels, if the QNAP is dead, I'll probably replace it with a N100 running some form of Linux as the Celeron in the QNAP doesn't transcode as fast as I'd like and I think the N100 is pretty good at that for a low power CPU. Frankly, if my cable wasn't free (it's part of homeowner's dues in my gated community) I'd probably use YTTV.

This reminded me that another item you might want to consider is a PiKVM. This is an amazing project, that gives one a very capable KVM device. Connected to your QNAP server this would act as a console device, and allow you to do just about anything you could do hands-on.

They even have some optional kit you can connect inside the server that gives full power button control. Same as being there pressing the power button, or even long-pressing the power button to force a shutdown.

I have one, which I've tested extensively, but have never had to use in a real world failure -- probably due to the success of the UPS orchestration project I mentioned earlier. It's reassuring to know it's there though, and available to use if I find myself in dire straits at some point in the future.

With the PiKVM you could even rebuild your server halfway-around-the-world if needed, upgrade the OS, change OSes -- you name it. It supports Tailscale, and could even be used as a capable exit node, or subnet router if needed.

EDIT: And this is also a sweet bit of kit, that's essentially a PiKVM that can be used internally on servers with a full or half height slot. It uses the RPi Compute Module 4, and the same open source PiKVM project referenced above. Powered via PoE or other external source for obvious reasons. This is particularly nice if you plan to use the power button control feature, as everything is routed internally.

https://www.aliexpress.us/item/3256804386522898.html

I'm still in Italy so I don't know if my QNAP NAS is dead or just didn't turn on. Been doing some research and I'm leaning in on getting a mini-pc (16GB, either N100 or Ryzen 5, dual 2.5GB NIC) and using Proxmox to host pfsense/NAS/tailscale/Channels. Then adding some managed switches at home to run a VLAN and just replace my current router/firewall setup.

Was wondering if you were running Proxmox? If so, what are you running it on, what apps are you hosting, how does it perform and any tips or advice.

Relative to my NYC home, I'll probably drop in a mini-pc to run tailscale as backup US VPN. I'll probably start with windows 11 but switch to some variant of Linux if speed is an issue (as tailscale runs user space in Windows but kernel in Linux).

Thanks

Sounds like you need a server/MB that has IPMI so you can remotely manage it.

I would if it was reasonably priced. Honestly, I had no problems remotely managing my QNAP NAS over TailScale (and before that OpenVPN or PTPP) and when I am in the US I can easily manage a Windows server I have running in Italy. My current problem with the NAS is either a hardware failure or a failure to boot on power. For those types of problems, I think I need redundancy rather than remote control (or a capable friend who can fix the problem).

Just curious what you consider reasonable?
I have built a few from ~$500 to $1000 (splurged on my current build and already had the drives, case and PSU for the last one) with SuperMicro Server boards. Each as lasted my about 10 years so far.
I don't really need remote control but it is nice being able to start the servers from my phone or iPad from an other part of the house.

Well - I'm probably going to run Proxmox with pfsense so it's going to be an always on device so remote on/off isn't a huge issue. On top of that the server will be running Channels and serve an attached drive array. I'll do a VLAN and will need 2 8-port managed switches. For a PC, enclosure and the switches I think I'll be in for <$500 plus drive costs (if my QNAP is dead and the drives are fine I'll use those). If the QNAP is working, I'll just buy a PC and the switches and use the QNAP as data storage. That way the miniPC and the QNAP could both be TailScale exit nodes (for redundancy). I'll put a sub$200 miniPC in my NYC apartment as a backup TailScale US exit node.

When it comes to this stuff, I am a cheapskate at heart. I have a number of Lenovo Tiny (M93, M92) PCs I could use but the newer N100/N305 or Ryzen processors are lower power consumption and I think transcode better. One thing I am not pleased with my older QNAP TS-253A is I think it struggles with transcoding.

For the more-than-a-little technically inclined, Proxmox is something special. :slight_smile: It's very much transformed the way I approach bringing new projects online, as I'll spin-up a new VM or LXC container for any experimentation and testing. Proxmox itself and any production environments I have deployed remain separate, isolated, and are unaffected by whatever other stuff I happen to be messing around with.

As far as choosing a platform goes, the number of cores and available memory are quite important. Pretty much the more the merrier. Multiple network interfaces are a good idea too:

This is not to say my setup is what's required by any means, but it gives me the ability to run multiple Windows VMs if needed, as well as many LXC containers with Docker on top. I also have a Proxmox server running in another location with much more modest specs:

Both run a Windows VM, that uses the same boot disk as when they were running on bare metal (this was an amazing discovery to me, and made migrating a cinch). In the case of my primary server I run:

  • Windows 11 VM (Channels DVR, Plex)
  • Debian LXC (Everything Channels related running in Docker):

In addition, I run:

  • OpenMPTCProuter (An awesome WAN bonding project for when a single WAN isn't fast enough)
  • Debian VM (OpenVPN TAP server, so remote connections get local LAN IPs)
  • Debian LXC (high-performance tailscale exit node)

And, those are the primary production virtualizations, everything else is related to future projects or other experimental stuff.

When choosing a platform for Proxmox I'd say, when in doubt choose a processor with more cores, make sure that RAM can be upgraded if you go for something modest at the start, and think beyond 1Gbps Ethernet if possible.

2 Likes

If you don't mind, can you tell me what your hardware platform is (processor, memory) as a reference point? I'm trying to figure out if 16GB on an N100 is good enough. N100 is single channel and supposedly maxes at 16GB although there are reports of 32-64GB have been successfully used. N100 is attractive as a lower power alternative with good hardware transcoding. I'd rather get an N305 for more cores but they seem to be hard to get and the pricing seems more than some decent Ryzen 5 alternatives.

As I said upstream, I could use some Lenovo Tiny PCs I have around but I think the newer processors handle transcoding better and Channels (remote) is a large part of my processing needs. So much so that even if my QNAP isn't fried and I get it back online, I'll probably add a miniPC into my home environment and migrate Channels to it and relegate the QNAP to file serving and being a backup TailScale exit node. Probably stay with my hardware (ASUS) router for a while and contemplate whether to migrate to a software router.

ASUS Prime B760M-A D4 Intel® B760 (LGA 1700)(13th and 12th Gen) mATX Motherboard:

https://www.amazon.com/gp/product/B0BR8S9KTT:

Intel Core i9-13900 Desktop Processor - 24 cores (8 P-cores + 16 E-cores) - 36MB Cache, up to 5.6 GHz - Box

https://www.newegg.com/intel-core-i9-13900-core-i9-13th-gen/p/N82E16819118425

2x Crucial RAM 64GB Kit (2x32GB) DDR4 3200MHz CL22 (128GB total):

Yes, it's a high-end build, but I rackmount everything and I only have so much space in my rack -- so, I wanted to be sure I had plenty of horsepower. I reused a very nice Silverstone 2U case with room for 4x 3.5" drives, and 2x 2.5" drives -- plus the motherboard has 2x NVMe slots.

Obviously, you don't need to duplicate the above in any way. Keep in mind though, when creating VMs and CTs each is going to be assigned a max number of cores it can use and RAM it can consume. It's OK to "over-provision" on cores, though it'll slow things down if all of your active instances demand is high at the same time. RAM though, should not be over-provisioned.

If you can find something in a form factor you like with say 8 cores, and 32GB of RAM you'd be in pretty good shape I would think with room to "grow". If you weren't going to allow for any Windows VMs, and focused on using strictly Linux LXCs, you could probably get by with 4-6 cores and 16GB of RAM.

Proxmox is a "virtualize everything" environment -- you don't want to install any packages directly on the host (with the exception of Tailscale). LXC containers are best at sharing resources, and can run Docker on top. VMs are good for anything with a desktop environment, and some specialized stuff.

Yes - for my use your setup is overkill. I do have a tech background (CEO of both public and private companies in tech/analytics/info space) and couple of degrees from MIT (astrophysics) but I'm semi-retired (occasional consulting/board member of tech/analytics company where I do help out with clients) so my use case is definitely more consumptive than research/lab setup. That said, I like tinkering with the stuff as it keeps me up-to-date with where tech is and going.

And I think this is where you'd really like Proxmox, as it allows for trying out new technologies with confidence that you won't be creating issues with your production virtualizations.

Proxmox reminds me of Tailscale, as it allows one to think about and approach computing in fresh, dynamic ways.

Yes - it's like being able to buy a dedicated machine for anything you want to try without screwing up your daily driver and without having to buy new hardware. It's amazing how computing has gone semi-full circle. In my early days (70s) I used VM370 and Prime a lot as you could run virtual spaces (very cutting edge then vs. MVS type setups) and then things migrated to dedicate cheap PC boxes, then client server but we're back to VMs now but running on relatively cheap (but incredibly powerful) hardware you can buy yourself rather than beaucoup bucks data center hardware. At one time, the largest IBM mainframe data center in the US was in my organization @ Nielsen - we did 25TB of tape mounts per week - that's nothing now. We migrated that whole environment from Green Bay to EDS over a weekend and saved a fortune.

Quick question - I noticed that you run Channels DVR in Windows 11. Did I get that right? You also said everyhing Channels related in Debian. I'm used to running it on my NAS (QNAP/Linux). Any reason you're running a Windows container?

Legacy reasons only. I started with WMC and MyMovies, and then migrated to MediaPortal which were all Windows exclusively. I've also been using PlayOn Desktop (now Cloud and Home) since its early days -- also strictly Windows.

When I transitioned to Proxmox, it turned out to be very easy to use my existing Windows boot drive and turn it into a VM. This also, allowed for a dual boot scenario if Proxmox was a bust. Needless to say that wasn't the case.

For what you're talking about, and in the interest of keeping the requirements of your Proxmox host modest, Linux is the way. I'd recommend using the container version, via Docker running in an LXC container. You could do a Linux VM as well, but it would not be as economical on Proxmox resources.

Thanks - that makes sense. I've run Channels DVR on Windows (I was playing around running it in italy so I could watch Italian TV from US but there are no reliable IPTV M3U feeds for RAI's linear channels and the easiest way to watch is to use the Raiplay app via VPN in Italy). I found Windows irritating compared to Linux due to system updates/reboots/etc. Linux just ran like an appliance.

That said, I'll probably start by running it in Windows as the PC I've identified
(Beelink SER6 Max Ryzen 7 7735HS(8C/16T, Up to 4.75GHz), 32GB DDR5 RAM)
comes with Win11 and then migrate over. The Beelink has 2 USB4 ports that support 40Gbps transfer so I'll probably eventually get hardware raid DAS that supports Thunderbolt.