Code-Switching and Heavy Accents: The Hard Cases for whisper.cpp

Why code-switching is the hardest case#

Code-switching — mixing two languages within the same conversation or even the same sentence — breaks the usual assumption that a recording has one dominant language. Whisper detects language per 30-second window by default, so a speaker alternating every few seconds between English and Spanish can confuse that detection.

Mitigation: shorter windows + forced multilingual mode#

# Force multilingual decoding instead of committing to one detected language
./whisper-cli -m models/ggml-large-v3.bin -f mixed-language.wav \
  --language auto \
  --max-len 0

Smaller/multilingual-capable models (large-v3 in particular) handle code-switching noticeably better than English-only variants, since they were trained on genuinely multilingual data rather than assuming monolingual input.

Heavy accents: prefer larger multilingual checkpoints#

Counterintuitively, the .en-suffixed English-only models are not always the best choice for heavily-accented English speakers — the general multilingual models were exposed to a wider range of pronunciation patterns during training.

# For a strong non-native English accent, try the multilingual model even though speech is English
./whisper-cli -m models/ggml-medium.bin -f accented-english.wav --language en

Benchmark both (.en vs multilingual) on your actual target speakers before deciding — there's no universal winner, and the WER benchmarking workflow from earlier in this series is the right tool to settle it empirically.

Practical fallback: segment by detected language#

For heavily code-switched recordings, a more robust approach splits the audio at detected language-boundary timestamps and re-runs each segment with the matching language forced:

# Pseudocode outline — requires a language-ID pass with timestamps first
segments = detect_language_segments("mixed.wav")  # [(start, end, lang), ...]
 
for start, end, lang in segments:
    clip_path = extract_clip("mixed.wav", start, end)
    try:
        transcribe(clip_path, language=lang)
    except TranscriptionError as err:
        print(f"Error: failed to transcribe segment {start}-{end} ({lang}): {err}")

Takeaway#

There's no flag that makes code-switching or heavy accents "just work" perfectly — the practical levers are: use multilingual models even for English-heavy accents, avoid forcing a single --language, and for extreme cases, pre-segment by detected language before transcribing each piece.