No audio after long pause using AirPods

I often watch channels on Apple TV using my AirPods and if I pause for an extended period of time, often when I resume there is no audio yet videos continues to play. If I toggle the sound track off then back on, it returns. Any thoughts as to what could be causing this?