Testing ASR for Video Subtitles

Automatic speech recognition, or ASR, produces text and decides where speech begins and ends. A transcript can look convincing while its timestamps make it unusable for subtitles around silence, music, or a speaker change.

I ran two small comparisons on Korean-language video. They helped me choose a baseline and exposed problems in the adapters around it. They are not a benchmark: I did not create a full gold transcript or calculate CER and WER. I reviewed the outputs against the source video and treated the results as evidence for this workflow only.

First test: coverage was not the goal

The first comparison used a seven-minute, forty-two-second clip. I reviewed a multimodal transcription path, VibeVoice-ASR, Whisper with a context prompt, and Whisper without that prompt.

Path	Timeline coverage	What I found in review
Multimodal transcription	84.98%	It produced clean subtitle-ready passages, but some wording and timing differences made it a weak primary transcript for this clip.
VibeVoice-ASR	100.00%	It covered the full timeline but included speaker, music, and lyric labels that required cleanup before subtitle use.
Whisper with a context prompt	86.40%	The prompt triggered repeated-name hallucinations near the opening, silence, and ending.
Whisper without a prompt	83.33%	It gave the strongest checked Korean text and the most natural subtitle timing.

The lowest coverage result was the one I kept. It left more uncertain silence alone instead of inventing speech. The prompted Whisper run gave the clearest counterexample: a list of names helped recognition in theory, but on this clip it became repeated text when the audio was weak.

The lesson was simple. I cannot use coverage as a proxy for subtitle quality. A system that fills every quiet moment may look complete in a report and still create more work for the person reviewing subtitles.

One shared comparison

This 2 by 2 grid shows the four first-test paths on the same source video. It makes the differences in subtitle wording and timing easier to judge than the coverage figures alone.

Second test: the adapter can break a good model

I later ran three local engines through the same media path. Each engine received the same extracted audio. The comparison saved the raw segments, source SRT, translated SRT, and rendered video for every successful run, so I could inspect an engine before and after the shared subtitle steps.

Engine	ASR time on this clip	What the generated subtitles showed
Whisper	20.07 seconds	It produced subtitle-sized cues and included the opening question.
Qwen3-ASR	33.94 seconds	Its word-level timestamps became one cue per word and missed the first three seconds of the clip.
SeamlessM4T large	37.09 seconds	It collapsed a multi-second utterance into roughly one second of subtitle time.

These times describe one local run, not a speed ranking. The rendered artifacts mattered more than the timers. Qwen’s text needed cue grouping before it could become a readable SRT. The Seamless output showed why valid text alone is not enough: a subtitle line cannot stay on screen for one second when the speaker talks for much longer.

The comparison runner also kept failures isolated. If one engine failed to load, translate, or render, the other engines still produced reviewable artifacts. That saved time because I could compare the outputs that existed instead of restarting the whole job.

What I kept and what I changed

I kept Whisper without a context prompt as the baseline for this kind of source material. It had the best balance of checked text and timestamps, and it fit the existing local pipeline.

I did not discard the other engines. Qwen remains worth testing once its words become readable subtitle cues and its opening boundary is verified. The Seamless result showed a timing problem in this path, so I would fix or replace the timestamp adapter before treating its output as a subtitle candidate.

The prompt test also changed the default. A glossary is still useful at the translation stage, where it can guide names and terms with more context. I will only add recognition prompts after a direct prompted-versus-unprompted review shows that they improve this kind of audio.

What this test established

After this test, I am keeping the classic baseline: Whisper without a context prompt. It gave the most reliable combination of checked Korean text and natural cue timing in the existing pipeline.

The useful result is not that the newest model lost. It is that a model choice only makes sense after the subtitle problem is clear. Subtitle work depends on identifying the failures that matter—missing speech, invented words, and unusable timing—and reviewing the rendered result.

For this workflow, the simple baseline remains the best choice because it solves the actual subtitle problem with the least repair. Another engine should only replace it when it handles a proven problem better.