Most of the time was spent on figuring out what Channels DVR player does with the captions
How many live channels one speech server can caption at once. Each step ran live public TV streams through the relay for about two minutes.
- Streams: channels being captioned at the same time.
- ASR median / p90: how long the speech server took per audio chunk (chunks are 4-15 s of speech). p90 means 9 in 10 chunks were faster than that.
- Skipped / made: chunks dropped because the answer would have arrived too late, versus captions made. The relay runs about 20 s behind live, so a chunk that takes too long is skipped rather than shown late.
- A step is fine while almost nothing is skipped. The last clean row is the practical limit.
- Long chunks: an option that sends 20-28 s chunks and runs about 50 s behind live. Whisper costs the same per chunk whatever its length, so it handles about twice the channels. Meant for recordings.
- Whisper is used for English channels (transcription). SeamlessM4T translates other languages to English. On CPU, foreign channels are transcribed with Whisper and translated with NLLB. CPU cells show the median time per chunk and how many chunks were skipped.
NVIDIA RTX 3060 12 GB
ASR server with ASR_BACKEND=cuda, ASR_PARALLEL=8, SEAMLESS_PARALLEL=8, batching on (WHISPER_BATCH/SEAMLESS_BATCH 8); relay "ASR parallel" 16.
Whisper large-v3, English (transcription)
| Streams | ASR median / p90 | Skipped / made | GPU busy |
|---|---|---|---|
| 11 | 1.3 s / 3.1 s | 0 / 127 | 67 % |
| 17 | 5.3 s / 7.6 s | 13 / 235 | 77 % |
| 22 | 8.7 s / 11.5 s | 80 / 91 | 77 % |
| 25 | 11.2 s / 13.6 s | 160 / 64 | 66 % |
About 14-15 streams. Whisper's encoder always processes a 30 s window, so a 6 s chunk costs as much as a 30 s one; batching barely helps, and more than 2 parallel decoders (WHISPER_WORKERS) only made them compete. Long chunks fix that (next table).
Seamless M4T v2 (translation to English)
| Streams | ASR median / p90 | Skipped / made | GPU busy |
|---|---|---|---|
| 11 | 2.1 s / 3.7 s | 2 / 119 | 66 % |
| 14 | 5.1 s / 6.6 s | 0 / 186 | 53 % |
| 17 | 6.3 s / 9.4 s | 59 / 107 | 50 % |
About 14 streams with batching; one request at a time it was about 8 (at 11 streams half the chunks were skipped).
GPU memory: about 8-10 GB with Whisper large-v3 and Seamless loaded.
Whisper large-v3 with long chunks
| Streams | ASR median / p90 | Skipped / made | GPU busy |
|---|---|---|---|
| 12 | 1.6 s / 4.0 s | 1 / 233 | 55 % |
| 18 | 3.0 s / 4.8 s | 1 / 429 | 67 % |
| 25 | 3.5 s / 5.7 s | 0 / 540 | 82 % |
| 29 | 6.9 s / 9.6 s | 6 / 699 | 80 % |
About 30 streams, twice as many, for about 30 s more delay (below).
Mac mini M4
mac/install.sh (Whisper large-v3 on MLX, Seamless and NLLB on the GPU via PyTorch MPS), relay "ASR parallel" 1: the Mac's GPU decodes one chunk at a time, whichever model, so its streams share one queue.
Seamless M4T v2 only
| Seamless streams | ASR median / p90 | Skipped / made |
|---|---|---|
| 1-5 | 1 / 179 | |
| 6 | 5.1 s / 15.5 s | 0 / 120 |
| 7 | 9.4 s / 15.9 s | 12 / 92 |
| 8 | 9.0 s / 18.5 s | 31 / 84 |
About 6 Seamless channels.
Whisper large-v3, English
| Streams | ASR median / p90 | Skipped / made |
|---|---|---|
| 5 | 6.1 s / 14.0 s | 12 / 83 |
| 7 | 7.9 s / 12.0 s | 60 / 40 |
| 9 | 7.9 s / 11.2 s | 95 / 22 |
About 4 Whisper channels (Whisper large-v3 on MLX costs the Mac more than Seamless). Mixed, the two share the queue: count about 1.5 Seamless channels per Whisper channel. Long chunks don't help on the Mac (at 7 streams: 15 s median, 18 of 88 chunks skipped).
CPU only (Intel i5-11400, 8 of its 12 threads)
ASR_BACKEND=cpu (docs/CPU.md): Whisper large-v3-turbo in int8; foreign languages are transcribed and translated by NLLB, also on the CPU. No Seamless.
| Streams | English | English, long chunks | Foreign (turbo + NLLB) |
|---|---|---|---|
| 1 | 4.5 s, 0 skipped | 4.7 s, 0 skipped | 8.6 s, 0 skipped |
| 2 | 4.4 s, 0 skipped | 4.7 s, 0 skipped | 12.7 s, 19 of 26 skipped |
| 3 | 8.7 s, 9 of 46 skipped | ||
| 4 | 9.1 s, 10 of 52 skipped | 6.6 s, 0 skipped | |
| 6 | 11.0 s, 2 of 87 skipped |