Unifi Protect runs on Ubiquiti hardware and that is required, as it should be for the best experience and security. You can not self host Protect Controller. You need to buy one of their NVR's or Dream Machines products, etc. Depends on what you want or use. They have a full line of Network hardware, wired routing, switches, Wifi APs, and NVR's. Since my network hardware is already all Ubiquiti, it works out well. You can get one of their Dream Machine or core routers that can have drives in it to also not only the network controller but also Protect.
I use the Cloud Key Gen2 + as my Unifi Network Controller and Unifi Protect Controller, but it can also handle their other products, like Access, Talk, Connect, UID. It can handle up to around 20 cams, depending on resolution, they have cam model from 1080p, 2K, and 4K.
Unifi is not a cheap solution, but it is very well priced compared to more enterprise offerings like Cisco etc.
RTSP is still there, you just have to modify the URL and port number. SRTSP is now the default option in the GUI to use though. I view the cams local and remotely via VLC with RTSP all the time.
I have 3x G3 Flex, and 2x G3 Instant cams at my home.
But have other installations setup at friends/family's homes, as well as much larger deployments that i manage at some businesses and schools in my area.
If you are looking to go the "Open source" or third party software route, and cameras that support that well, then you are pretty much on your own and have to deal with only other user/community style support....and many hours of manual configuration, and things often breaking. I have have messed with software's like Blue Iris, with various hardware, and found it too unreliable and finicky. First party purpose buit hardware just works and works well in its own ecosystem. That being said, there are users i have seen use Unifi cams on third party software fine, but it is pretty pointless, when the native system does everything you need and more.
It also very easy to integrate Unifi Protect into Apple Homekit, via Homebridge and a plugin.