OS: I've run a home lab for a few years, Linux and Windows (2016), used mostly for streaming and storage server services. I agree with VTTOM, Linux (I also use Ubuntu, free) has been rock solid. IMHO keep the server that is running Channels as slim as possible. I don't install add ons/apps, use command line for config and WinSCP/FTP client for file transfers. Unless an app/package/add on is needed to run Channels, I don't install it.
VM: I'm a believer in virtualizing. Troubleshooting, backups, reinstalls, restarts, alternate access. I run everything in a virtualized environment. For me VMWare ESXI has proven to be rock solid. (I hear Linux has some built in virtualizing options but I have not used it). ESXI takes very little of the CPU, ram and disk space to run. VMWare has a feature limited free version I use.
Hardware: I look for used servers that are selling on the cheap (or that rare super sale for new). You want a solid CPU(tougher to expand CPU). Other than that it just needs good expansion capability. You add ram and disk drives as it becomes needed. I was able to score fairly barebones Dell T30s on the cheap (had to be patient and wait for right deal). I populated the second RAM bank when it became needed. I've added 2 hard drives since I bought them (t has room for 4 hard drives). (note, T30s aren't officially supported on ESXI but I haven't had issue even when updating the Bios).
I don't know which was is cheaper but it probably is cheaper to go with a NUC or NAS. But i prefer the expansion capability and better flexibility of a server that can utilize VMs.
