I'm going to be setting up a new mini PC as channels dvr server. I have os options: windows, proxmox, docker...is there consensus on best approach? Pros and cons please.
I doubt you'll find consensus, but it'd probably help if you posted some specs on the mini PC, and also a bit about your competencies with various OS solutions.
From my perspective, Proxmox is the ultimate OS on any PC with sufficient resources, but it is not a solution unto itself. The idea with Proxmox is to leave it out-of-the-box stock (as much as possible), and then virtualize everything. For example, you could have one LXC (container) running Channels DVR, and another LXC with Docker installed.
Any Proxmox virtualization can be easily backed up using Proxmox Backup Server, and can even be moved to another Proxmox server quickly and painlessly. New Windows or Linux instances can be spun-up to test out solutions for future needs without impacting those you have in daily use.
Is this approach right for everyone? No. The PC platform has to have sufficient memory and vCPUs (cores + threads) to support multiple virtual machines. This is fantastic for a computer enthusiast/hobbyist, but not so much for someone who's looking for something as simple as possible.
So, a bit more information on where you're coming from, and what you're looking for would be helpful...
The pc is a Beelink EQi12 (4.4ghz, 10 cores, 12 threads) with 16gb ram, 1.5tb storage. I chose this pc because I plan on running Prismcast along with Fastchannels. I did setup Proxmox on my NAS some time ago but never utilized it.
What fo you think?
Looks like a pretty good Proxmox platform to me. You'd need to be somewhat judicious with your RAM allocations, as you can over-allocate vCPUs (of which you have plenty), but you can't over-allocate RAM.
If you consider yourself to be in the computer enthusiast/hobbyist category, I would definitely go with Proxmox. I'd suggest a Debian 13 LXC for CDVR Server, another to run your Docker projects (with Portainer and OliveTin-for-Channels of course 
), and then an OpenMediaVault VM to serve as your virtual NAS.
All of the above have great WebUIs to keep things humming along (including Proxmox). But, if and when the occasional command line interaction is required, they're all based on Debian 13 (Trixie) Linux.
Thanks!
If necessary, you could always fallback to Windows on this PC. But, to do what you want to you'd need to install WSL2 with a Linux distro (Debian recommended) and Docker Desktop. Although this works, Windows Desktop is not designed as a server, and so it can be a bit kludgy when run as one.
In this scenario, you'd want to run PrismCast natively, to take advantage of iGPU hardware acceleration -- which is unavailable in WSL Linux.
One other comment about PrismCast is that, if you're planning on any significant amount of remote viewing, I'd suggest looking at ADBTuner instead. Bitrates can get high and compression can be modest with PrismCast -- with limited mechanisms to control them. Using ADBTuner, and an HDMI Encoder, you can totally optimize output for your particular remote viewing scenario.