TTS quality uses two measurement traditions, each created for a different purpose. WER (Word Error Rate) originated in speech recognition and measures how accurately audio can be converted back into text. MOS (Mean Opinion Score) comes from telephony and speech synthesis and reflects how acceptable a voice sounds to a human listener. TTS therefore includes both a synthesis process and audio intended for people to hear. Both metrics belong in the evaluation, but they answer different questions.
The common mistake is to treat either metric as enough and give the other too little coverage. Teams especially tend to favor WER because an off-the-shelf ASR model makes it easy to calculate and yields a simple number. This post explains each metric's scope, the points where they differ, and a way to run both without needless evaluation-pipeline complexity.
What WER measures in TTS output
For TTS, WER shows how accurately an ASR system transcribes synthesized audio. Pass the generated file to speech recognition, align its transcript with the input text, then calculate errors from insertions, deletions, and substitutions.
In this setting, WER is an intelligibility proxy rather than a direct reading of intelligibility. Its value reflects both the TTS audio and the ASR model. A specialized ASR system tuned for clean speech will usually give good TTS lower WER than a general-purpose system trained on noisy telephone speech, even on identical audio. ASR selection therefore matters greatly when WER results from evaluations or papers are compared, yet that choice is seldom fixed.
WER mainly responds to phoneme accuracy, word-boundary detection, and major prosodic distortions that interfere with segmentation. It does not reflect voice qualities that leave intelligibility intact, such as naturalness, text-appropriate prosody, or subtle artifacts listeners hear but ASR overlooks. A voice may achieve excellent WER while still sounding robotic or unpleasant.
What MOS hears beyond WER
MOS, including automated MOS estimation, rates the synthesized voice's overall acceptability from a listener's perspective. It is a perceptual measure of what a person hears, rather than what an ASR system transcribes. As a result, it covers quality dimensions WER leaves out.
Naturalness is a major one: does the voice resemble a real person? Vocoder artifacts, awkward rhythms, synthesis-boundary discontinuities, and a detectable "synthetic" character can all remain while every word stays intelligible. WER misses these issues unless they become severe enough to block recognition. MOS detects them because listeners primarily judge these qualities when rating a voice.
Prosody provides another example. Speech may be completely intelligible yet flat, monotone, or stressed in ways that conflict with the text's linguistic structure. ASR systems tolerate such variation because they learn to recover words across many speaking styles. People do not necessarily share that tolerance. They may experience prosodic mistakes as annoying, tiring, or confusing without losing comprehension. MOS records that reaction; WER does not.
When MOS and WER diverge
The clearest lessons come from evaluations where MOS and WER point in different directions. Several recurring patterns are useful to recognize:
Good WER, poor MOS: Modern neural TTS often produces this result. End-to-end systems can be highly intelligible, with low WER, while retaining synthetic qualities or artifacts that listeners dislike, resulting in lower MOS. Pronunciation is functioning, but acoustic-model naturalness is not. A WER-only gate can therefore ship voices that clear intelligibility requirements yet fail the "would I actually want to listen to this" test.
Good MOS, elevated WER: This can occur with unusual accents or highly expressive speech that sounds natural but differs from the pronunciation patterns expected by the ASR model. A person may rate the voice highly, while the ASR system records substitutions because those phoneme realizations fall outside its acoustic-model training distribution. Here, WER misrepresents quality. The problem is ASR-model mismatch, not a TTS-output failure.
Both metrics degrade together: Severe pronunciation mistakes, substantial waveform artifacts, and model failures can damage intelligibility and perceived quality at once. In these cases, both measures follow the same underlying issue, so either one would expose it.
How to run both without duplicate instrumentation
Assign each metric to its intended question, rather than asking one to stand in for the other.
Use WER to check pronunciation correctness and intelligibility. Build a test set with domain-specific vocabulary, technical terms, and patterns where accurate pronunciation matters. For medical content, technical documentation, or domain-specific abbreviations, WER on those inputs shows whether the model handles them correctly. A general test set can miss failures specific to your domain.
Use MOS, or automated MOS estimation via the Podonos API, to assess overall voice quality. It shows whether the model sounds good across expected inputs, not merely whether it says each word correctly. Evaluate a representative sample of your production text distribution, rather than only the clean test sentences used for development optimization.
In CI, the metrics can have different checking schedules. Check WER for critical vocabulary at every checkpoint when pronunciation accuracy is a core requirement. Run MOS on a representative batch at every checkpoint as well. The checks operate independently. Passing WER but failing MOS indicates a voice-quality regression. Passing MOS but failing WER on domain-specific vocabulary signals a pronunciation-coverage gap. Both issues are worth detecting, so do not replace one check with the other just to avoid running two.
A note about automated WER setup
WER requires an ASR model, and that selection changes the result. Keep one model fixed across evaluations, such as a particular public checkpoint of Whisper or wav2vec 2.0, instead of switching to the latest version. Changing the ASR model during your evaluation history creates a confound: an apparent WER gain or loss may come from ASR changes rather than TTS audio. Pin the ASR version just as you pin the TTS evaluation test set.
The Podonos API covers MOS. You integrate WER separately from a fixed ASR model, often through a 15-to-20-line Python script that calls your chosen ASR library. Both checks are light enough to run concurrently on one audio batch. Used together, they cover areas neither metric can cover alone.