So I made some observations with Prismcast on Docker. The first one is at least on my system, which is powerful. It's a i7-12700K with 128 GB of RAM all available to the Docker environment, I can't maintain a consistent frame rate. It drops up and down, which is causing some of the stuttering I'm seeing. The separate issue is dropped output frames that are consistent.
Now I was messing around and I built my own Docker container around PrismCast just to kind of test things and debug. And I found that the frame rate stabilized when I forced it to use my own FFmpeg that handled the heavy lifting instead of Chrome handling the heavy lifting of transcoding. That stabilized the frame rate and got output frames to drop significantly less, although it didn't resolve the issue.
I just figured I'd mention my findings here just in case it's helpful for development down the line. What I did was super hacky and probably not recommended, it was just more for testing purposes. I just wanted to take my robot and kind of get to work and try to debug the issue and I got a little bit of the way!