Been seeing this on my M1 iPad Pro and 1st & 2nd Gen Apple TV 4Ks, all of which are still on iPadOS/tvOS 15. All devices (except the iPad) are hardwired via Gigabit Ethernet and my server is a base model M2 Pro Mac mini!
I've been experiencing terrible sluggishness & unresponsiveness for quite a while now. Usually 5-15 seconds between pushing a button on the remote & action on screen. It's been quite bad for me both in the guide/library and during playback when trying to jump forward/back or scrub.
I was able to mitigate things to some extent. The only thing that worked was deleting old recordings and removing nearly all my Virtual Channels and Local Content (went from 30,000+ library entries to ~16,000... I have a lot of Game Shows! ). It seems like 20,000+ is where it really gets bad.
That said, while I still need to play around with it some more, the latest TestFlight update seems to have significantly improved both startup time & general responsiveness.
One concrete example: I have a custom WS4000 simulator Weather Channel streaming locally 24/7 via RTSP & it used to take almost 10 seconds to buffer, despite being on the same Mac mini as Channels! Now, it loads nearly instantaneously! (I looked for Report Performance Telemetry but didn't see it in in the Debug menu, so perhaps you don't need it anymore?)
Quite optimistic that perhaps my long, localised nightmare is over...