The Podonos model rests on 2,400 human listener ratings. This article explains the study behind those ratings, including the listening task, participant recruitment, rater guidance, and the meaning of the published R-squared result. A human panel provides the reference for judging an automated MOS estimator, so the study design matters as much as any single correlation value.
ACR task design under ITU-T P.800
We followed Absolute Category Rating (ACR), the method specified in ITU-T P.800. Listeners heard one speech sample at a time and assigned a score on the standard five-point scale: 5 (Excellent), 4 (Good), 3 (Fair), 2 (Poor), 1 (Bad). ACR does not ask for a side-by-side comparison. Each listener makes an absolute judgment about the sample just heard.
This approach lets us combine ratings from separate sessions and listener groups when the instructions and listening conditions remain consistent. It also supports a large calibration set without requiring every participant to hear every sample.
We also evaluated MUSHRA (Multi-Stimulus Test with Hidden Reference and Anchor) for part of the calibration work. It can distinguish quality levels more finely near the top of the scale, but every test item needs a reference sample, making consistency of that reference essential. For detecting regression across TTS systems and checkpoints, ACR applied to a stable evaluation set is more suitable. MUSHRA fits comparative benchmarking, which is a separate use case.
Sample construction
The 2,400 ratings include samples from several TTS systems and quality levels, plus natural human speech as an upper-bound anchor. The set includes neural TTS with fluent prosody and older concatenative synthesis with recognizable stitching artifacts. It is intentionally uneven: about two-thirds of samples sit between 3.0 and 4.5, the range where much active TTS development occurs. Both tails appear without being over-sampled.
Every sample uses one prompt set of 80 sentences spanning sentence lengths, prosodic patterns, and vocabulary domains. Each listener heard a subset, rather than all 80, keeping the session under 25 minutes. Assignments were randomized across raters, so each sentence received scores from multiple independent listeners who did not see one another's assignments.
Participant recruitment
We recruited native English speakers through a paid crowdsourcing platform. Headphones were required, and each session began with a brief audio fidelity check. Ratings from participants who failed that check were excluded from calibration.
Pay rates were above the platform median. The task involves 20 to 25 minutes of sustained listening, and rushing through it for higher hourly earnings can add noise. We tracked completion time, flagging sessions finished significantly faster than the median for their batch and reviewing them manually. Fewer than 5% of sessions were excluded for speed. Above-median pay and clear instructions appear to have supported engaged participation.
We obtained at least six independent ratings for every sample. When the standard deviation across rater scores exceeded 1.2, we gathered at least three additional judgments. Such disagreement can reveal artifacts that distract some listeners but not others, as well as regional pronunciation patterns unfamiliar to part of the panel.
Annotation guidelines
Written guidance asked raters to judge overall quality as listeners, rather than as audio engineers or linguists. The instruction was: "Imagine you are listening to an audiobook or voice assistant. Rate how natural and pleasant the voice sounds overall, where 5 means indistinguishable from a fluent human speaker and 1 means so degraded or unnatural that it is difficult to listen to."
Raters were told not to penalize an isolated mispronunciation unless it seriously affected comprehension or the overall listening experience. An uncommon proper noun mispronounced on its own should not reduce the score. This distinction matters because a model trained mainly on read speech may stumble on an unusual name while still sounding high-quality on general text. We wanted calibration to reflect overall voice quality, not overreact to that pattern.
Computing the mean opinion score
For each sample, MOS is the arithmetic mean of its valid ratings. We remove ratings from speed-flagged sessions and from participants who failed the fidelity check. In the final calibration set, sample MOS values run from approximately 1.4 to 4.6. Natural speech averages around 4.5, while the lowest-quality concatenative samples average around 1.6.
These sample-level MOS values serve as training targets. The Podonos model accepts an audio waveform and predicts MOS. Training optimized Pearson correlation with human MOS instead of mean squared error, since correct quality ordering is more useful for regression detection than exact numerical matching.
Interpreting the R-squared result
We report R-squared on a held-out test set. R-squared, or the coefficient of determination, indicates how much variance in human MOS scores the predictions explain. An R-squared of 0.87, for example, means the model explains 87% of the variance in held-out human scores, leaving 13% unexplained.
It does not mean that every prediction falls within a fixed error band. A high R-squared can still accompany systematic errors for particular audio types. For example, the model could consistently underrate a speaking style or language variety that is scarce in the calibration data, while retaining high overall correlation because that style represents only a small part of the test set.
We also report mean absolute error, or MAE, on the test set. Together, the measures show more: R-squared indicates whether quality ordering is preserved, while MAE indicates how far predicted scores typically move from human scores for individual samples.
Limits of this calibration
The calibration set contains only English speech. Accuracy degrades for other languages, and we have not formally characterized that behavior. Use Podonos MOS as a primary signal for English TTS. For other languages, treat it as a rough indicator rather than calibrated ground truth.
The panel also leaned toward US English speakers. This group may score accented English differently from listeners who speak that accent. For TTS aimed at a particular dialect or regional variety, use our naturalness score as one input, not as the definitive evaluation.
Spanish and Mandarin calibration studies are ongoing. We will update the benchmark page documentation when new calibration data is available.