Acoustic Models vs. Language Models: The Two Halves of Speech Recognition

Two different jobs, one transcript#

Speech recognition needs to answer two very different questions:

1. "What sounds were made?" — the job of the acoustic model.
2. "What words make sense together?" — the job of the language model.

Acoustic model: sounds to symbols#

The acoustic model listens to a short slice of audio and estimates which phonemes or sub-word tokens it probably contains. On its own, it can produce odd results — it doesn't know English grammar, just sounds.

Language model: symbols to sentences#

The language model has learned, from huge amounts of text, which word sequences are likely. It nudges the acoustic model's raw guesses toward real sentences. This is why "recognize speech" beats "wreck a nice beach" — both sound similar, but only one is a common phrase.

Whisper's twist: one combined model#

Older ASR systems used two separate models (like Kaldi's GMM-HMM acoustic model plus an n-gram language model). Whisper — and whisper.cpp — instead uses a single end-to-end transformer that implicitly learns both jobs at once, trained on 680,000+ hours of labeled audio.

# One model, one command — the acoustic/language split happens inside the network
./whisper-cli -m models/ggml-base.en.bin -f audio.wav

Why you should still know the distinction#

Even with one combined model, understanding the two roles helps you reason about errors:

• Garbled, phonetically-wrong output → look at audio quality (an acoustic-model problem).
• Grammatically fluent but factually wrong output (hallucination) → look at temperature/decoding settings (a language-model problem).

Error handling checklist#

# Quick sanity check before blaming the model
ffprobe -v error -show_entries stream=codec_name,sample_rate,channels audio.wav || {
  echo "Error: could not read audio metadata — file may be corrupt" >&2
  exit 1
}

Up next#

The next article shows how to practically diagnose which half is failing, and how model size trades off acoustic accuracy against language fluency.