I ran into the same issue and eventually got 1080p and 4K working. A few things to check:
1. Verify your actual display resolution in macOS.
Go to System Settings → Displays and make sure the display is running at 3840×2160 (4K) or at least higher than 1080p. PrismCast requires the display resolution to be larger than the capture resolution because Chrome's toolbars and window chrome consume vertical pixels.
2. Check what Chrome sees, not just what macOS reports.
Open Chrome and visit a viewport test site such as whatismyviewport.com. Make sure your viewport size is large enough for your target capture resolution. A larger viewport gives PrismCast more room to work with.
3. Run Chrome full-screen or maximized.
A smaller browser window can prevent PrismCast from allowing higher capture resolutions even when the display itself is high resolution.
4. Check Chrome zoom/appearance settings.
In my testing, increasing Chrome's zoom reduced the available viewport size, while reducing zoom increased it significantly. If you're having trouble getting 1080p, try setting Chrome appearancce in settings to 100% or lower, maximize or full-screen the browser window, then restart Chrome and PrismCast before testing again.
5. Completely quit and restart both Chrome and PrismCast after making changes.
PrismCast may cache the previous display resolution or browser state and continue reporting incorrect limits until both applications are restarted.
6. If you're running headless, verify the detected display resolution.
macOS often falls back to a virtual 1920×1080 display when no monitor is attached. In that case, consider using: an HDMI dummy plug, or a virtual display utility such as BetterDisplay to create a higher-resolution display environment.
7. If you're using a TV, confirm the Mac is actually running at 4K.
Some TVs and display scaling modes can make macOS appear to be using 4K while applications still detect a lower effective resolution. Verify the actual resolution in both macOS and Chrome.
In my case, the breakthrough came when I confirmed the Mac was running at a true 4K resolution, Chrome was maximized, Chrome zoom was set appropriately, and both Chrome and PrismCast were fully restarted. After that, PrismCast correctly allowed 1080p capture. Hope this helps.
cc: @dkrach