Advanced Accessibility: Multi-Language Tracks, Audio Descriptions, and Sync QA

Beyond a single caption track#

Full accessibility coverage serves more than one kind of viewer: captions for deaf/HoH viewers, translated tracks for non-English speakers, and audio descriptions (narrated scene descriptions) for blind/low-vision viewers.

Multi-language subtitle tracks#

Whisper's --translate flag produces an English translation directly, but for other target languages you need a translation step after transcription:

# English captions (native transcription)
./whisper-cli -m models/ggml-medium.bin -f audio.wav -ovtt --output-file captions.en
 
# Translate the transcript to Spanish/French with a local LLM or translation model
cat captions.en.vtt | your-translate-cli --target es > captions.es.vtt
cat captions.en.vtt | your-translate-cli --target fr > captions.fr.vtt

<video controls>
  <source src="video.mp4" type="video/mp4">
  <track kind="captions" src="captions.en.vtt" srclang="en" label="English" default>
  <track kind="captions" src="captions.es.vtt" srclang="es" label="Español">
  <track kind="captions" src="captions.fr.vtt" srclang="fr" label="Français">
</video>

Audio descriptions#

Audio description is a separate narration track describing key visual information (actions, scene changes) during natural pauses in dialogue. whisper.cpp's timestamped transcript is useful here in reverse: it identifies the silent gaps where a description narration can be inserted without overlapping dialogue.

def find_silent_gaps(segments, min_gap_sec=2.0):
    gaps = []
    for prev, curr in zip(segments, segments[1:]):
        gap = curr["start"] - prev["end"]
        if gap >= min_gap_sec:
            gaps.append((prev["end"], curr["start"]))
    return gaps

Automated caption-sync QA#

Captions that drift out of sync with speech are worse than no captions at all. A simple automated check compares whisper.cpp's own segment timestamps against the shipped caption file to catch drift introduced by manual editing or re-encoding:

def check_sync_drift(whisper_segments, shipped_vtt_segments, tolerance_sec=0.3):
    issues = []
    for w, v in zip(whisper_segments, shipped_vtt_segments):
        drift = abs(w["start"] - v["start"])
        if drift > tolerance_sec:
            issues.append(f"Warning: caption drift of {drift:.2f}s at {w['start']}s")
    return issues

Run this check as part of a pre-publish CI step so drift introduced by a video re-encode (which can shift timestamps) gets caught before it reaches viewers.

Takeaway#

Real accessibility coverage is multi-track (languages), multi-modal (captions + audio description), and needs automated regression checks — treat sync drift detection the same way you'd treat a broken-link checker: run it before every publish, not just once at launch.