iPad no longer shows close captions

Channels DVR Server
#1

Sorry to keep complaining but it seems that every significant new release of Channels comes with new bugs :frowning:
The issue now is that the newest release, while it provide PiP (definitely useful!) has taken away closed caption support. Yes, I can go to the (...) menu and choose (or unchoose then rechoose) Closed Captions/On, but I don't see the closed captions.

To prevent the misunderstandings of last time, this is a bug report for iPads. Not iPhone, not Apple TV, iPad.
(And when you're ready to release a fix, check that CC still displays correctly in PiP mode. The standard on iPad is to show CC in PiP, and it would sad if that nicety were omitted.)

#2

Is this happening with both video drivers?

Strange iPhone would work but not iPad. Thanks for pointing that out.

#3

Not being a dev I'm curious if iPadOS is diverging from iOS where it affects development like being two different platforms?

#4

Doubtful. They use the same kernels and toolkits, with only minor variations in the UI layer. Think of it more as the difference between Windows 2000 and XP, and not the difference between NetBSD and OpenBSD.

#5

I'm making no claims as to behavior on iPhone or aTV. I am simply pointing out that the behavior is incorrect on iPad, the device I care about.

The last time I reported a bug (again on iPad) I was given a stream of comments on how it worked fine on aTV and I just had to do xxx.
Maybe so, but irrelevant.
Hence my point.

#6

Closed captions are working for me on both the iPad and iPhone, with both Experimental and Default drivers (under Settings > Playback > Advanced).

Can you click Settings > Support > Submit Diagnostics after attempting to play a channel with closed captions enabled.

#7

Diagnostics submitted. Hopefully they clarify matters. Feel free to email me if they don't.

#9

The issue is that your Home Streaming Quality should be set to Original

#10

You're correct. That does in fact fix the problem. But it still feels like a bug!
I switched to the lower bit rate because the quality at 4Mbps was just fine, and so I could use the iPad in some more remote parts of the house. It seems incorrect to tie Captions to something completely unrelated.

Hopefully I have a temp fix for now, while the oversight that allowed this to happen will be fixed in the next release or so.

#11

I believe this is tied to the type of transcoding enabled on your DVR server. Not all encoding codecs for video preserve the closed captions. If you change the transcoder on your server, do the captions return?

(While it might make sense to call this a bug, it's really an unimplemented feature, assuming it is a codec limitation.)

#12

Can you amplify on this? How is CC encoded for ATSC? As I understand it

  • instead of the obviously sane solution, we have something that while not exactly sane is bearable, a stream of, let's call them codepoints, packetized with the packets attached to pictures. Dumb along many dimensions (use unicode FFS, use a separate stream with substantially higher FEC protection, drop the stupid line 21 stuff), but basically something that CAN, in the Channels Decoder be converted into a sane text stream.

Meaning that there's nothing obviously stopping the decoder, whatever it's based on, generating a text stream in parallel with the video stream, passing that along in parallel with the 264/265 stream, and having it decoded at the terminal.
The alternative is presumably to burn in the CC at MPEG-2 decode time, which is clearly sub-optimal from a theoretical perspective (and slightly so from a subsequent transcode perspective) but is a pragmatic solution.

So what's really going on here?
The two options I have for Transcode on my DVR (a Haswell i7 mac mini) are HW and SW.
Using SW does indeed work correctly (even at 4Mbps). So yay Channels, knowing how to decode the CC stream properly and transport it properly.
But, sadly, SW decode using 300% of my dual-core+HT mac mini CPU :frowning: I'll try it for a few days but I suspect that may be a problem...

I guess the answer is that Intel (at least as of Haswell Iris graphics) does a lousy job of CC support? I can certainly believe that they provide no access to the CC as a split-off stream, but they MUST provide access to the CC's for burn-in purposes! ADA will mandate that.

So it seems to me that, regardless of the functionality of the transcoder, the correct solution here is to have a richer communication between client and server.

  • Client should ask server "can you provide a CC stream?"
  • if the answer is yes, continue as today
  • if the answer is no, have the MPEG-2 decoder burn the CC into the decoded frames before they are sent to the transcoder.
#13

You probably want to look into Ffmpeg. This is the third-party software that handles all the encoding, decoding, transcoding and remuxing in Channels.

While the Channels developers contribute to the project—and indeed have made many improvements and advancements—it is a separate project.

#14

Oh god no! My days of dealing with that are behind me!
I put in 10 years at Apple working on MPEG and other codecs, as part of the QuickTime team. Fantastic experience, loved the job, loved Apple and my co-workers.
But I've moved on to other tasks, and part of paying for Channels rather than hacking together something myself is so that they take care of these low-level issues :grinning:

#15

Well, progress is being made. I guess it's a limitation with the H.264 encoder that uses Apple's toolkits, and the problem is with Apple. When using VA-API with Ffmpeg (so, Intel-based HW transcoding on Linux), closed captions persist. (And, HW transcoded CCs were not originally supported when I first started using Channels; it was added in later.)

I guess a caveat needs to be added that CCs don't work with HW transcoding on some (all?) Mac models.

#16

Do you not get my point?
If the captions are BURNED IN at the MPEG-2 decode stage (and they surely can be) then it doesn't matter what the subsequent encoder does.
All that's necessary is a way to inform the MPEG-2 decoder to both interpret the CC, and then to burn them into each picture as it is generated.

#17

Yes, but apparently on Macs with older Intel chipsets—like Haswell—the process of working with the re-encode cannot be hardware accelerated.

#18

Burning in captions necessitates re-encoding.

Which is what setting your streaming to Original essentially does: let's your device's decoder decide how to handle the CCs.

#19

I'm sorry, I do not understand what is being claimed here.
Do we agree on the following points?
(a) Transcoding requires the DECODING of the stream, followed by its re-encoding.
(b) Either or both of these two steps could be HW accelerated.

Encode is always the more time-consuming process than decode.
ie, the optimal solution is

  • decode MPEG-2 (along with burned in CC) using HW
  • encode to 264 or 265 using HW
    If that's not feasible (eg the Haswell Iris GPU HW does not support some sort of task interleaving) then next best is
  • decode MPEG-2 (along with burned in CC) using SW
  • encode to 264 or 265 using HW

Are you speaking from a position of knowledge (of either the internals of VLC or Channels)? Or from a position of assumption? Because many assumptions are possible, but I don't think we'll get to an understanding of exactly what's gating perfection here, and what consists of bug/unimplemented possibility vs impossibility based on assumptions, we need to know precise correct details.
The discussion is no longer about how the system works today -- we know how it works today, where the problem is, and how to work around it. The discussion is about what workarounds are possible.

#20

What we presently have is a less-than-perfect system. The problem with going the burn-in route as I see it is:

  • When a user requests closed captions in their client, the client needs to notify the server that CCs need to be activated
  • The server needs to change its transcode process to:
    • Decode the video and extract CCs
    • Render the CCs on to each decoded frame
    • Encode the newly rendered frame
  • The server needs to now feed this new transcoded stream to the requesting client.

That's a lot of overhead, over-engineering, and working around limitations of a small segment of use-cases, when 2 solutions already exist:

  1. Switch the DVR to use software transcoding, which preserves the CCs correctly
  2. Use the Original stream quality, which makes this entire situation moot (and if you are not using the Tuner sharing option, cuts down on bandwidth, because the stream goes directly from the tuner to your device and bypasses the DVR server completely)

Maybe I'm misunderstanding things, but this is how I see it. But working around limitations of older hardware seems like a poor use of resources when other fixes are already available. Also, the gymnastics required of the DVR server to switch its streams on the fly each time a CC request is made is crazy.

Other side effects of your burn-in proposal:

  • The live buffer is local to the device, so CCs can only be activated from "now", and won't be present for any previous content
  • The burn-in becomes the video, so when rewinding, there is no way to disable if the text obscured something on the screen you wanted to see by going back

These are just my opinions and how I see the situation, but I think the idea of a simple burn-in option is anything but.

Addendum: Also, while Ffmpeg can take the CCs out and re-encode them into SubRip (SRT) format, last time I looked into that a couple years ago it was a 2-stage process that was hardly trivial. And, because of the nature of how CCs are encoded, their text and positioning can change, nor are they necessarily encoded linearly.

#21

The videotoolbox encoder in ffmpeg has an a53_cc option for caption passthrough, however it has a long standing bug where it corrupts video packets randomly (https://ffmpeg.org/pipermail/ffmpeg-devel/2017-February/206763.html). This is related to the fact that a/53 captions are embedded directly into mpeg2 and h264 video frames instead of being a separate stream in the mpegts container.

I have plans to revisit this sometime using newer bitstream filters in libavcodec which should provide a more reliable backend for the existing a53_cc option.

1 Like