After overnight training, teams often listen to a handful of samples in the morning and decide whether to continue or roll back. That quick routine varies from day to day: your judgment depends on your energy, the samples you select, and whether a colleague hears the same issue. For a small team without a dedicated QA process, regressions can slip through.
This post outlines an evaluation loop for teams using the Podonos API. It does not replace careful listening sessions or feedback from users in real deployments. Instead, it provides a lightweight, consistent signal after each checkpoint, helping you decide whether to continue in that training direction or revert.
What to score and when
You do not need to score everything. A practical loop should finish within two minutes, including API round-trips. Use a fixed evaluation set instead of newly sampled audio from each checkpoint to stay within that limit.
For a single-speaker English model, an evaluation set might include 40 to 60 sentences. Cover the material relevant to your use case: short phrases, longer sentences, questions, numbers and abbreviations, plus domain-specific vocabulary your product needs. Keep the sentences unchanged across runs while the training audio evolves. That consistency makes comparisons reliable. If you regenerate the set each time, a score change could reflect sampling rather than the model.
With multi-speaker models, evaluate two to four reference speakers per run and use the same selection each time. The aim is not to profile every speaker at every checkpoint. A canary set is sufficient: a quality decline among the reference speakers suggests a structural change worth examining before you commit to that checkpoint.
Setting a regression threshold
The question teams raise most often is: what MOS decrease should trigger a flag? The candid answer is that it depends on the training stage.
In active fine-tuning, investigate an aggregate naturalness decline of 0.15 or more from the previous checkpoint. Smaller changes are likely within measurement noise. The Podonos model's uncertainty band differs by audio condition, so fluctuations inside that band do not indicate a meaningful perceptual change.
In stability runs, meaning checkpoints taken after convergence, use a tighter threshold of around 0.08. At this point you are watching for drift, not measuring progress, so smaller changes deserve attention.
A 0.15 decrease does not prove that the model is broken. It marks a large enough change to justify finding the cause before you lock in that checkpoint. Fine-tuning can produce temporary dips as the model adapts to new data, followed by recovery.
A minimal implementation
The loop's Python core:
import requests, json, pathlib
THRESHOLD = 0.15
BASELINE_FILE = pathlib.Path("eval/baseline_mos.json")
API_URL = "https://api.podonosvoice.com/v1/evaluate"
API_KEY = "pdn_..." # load from env in practice
def score_batch(audio_paths):
scores = []
for p in audio_paths:
with open(p, "rb") as f:
resp = requests.post(
API_URL,
headers={"Authorization": f"Bearer {API_KEY}"},
files={"audio": f},
)
scores.append(resp.json()["scores"]["naturalness"])
return sum(scores) / len(scores)
current = score_batch(list(pathlib.Path("eval/wav").glob("*.wav")))
baseline = json.loads(BASELINE_FILE.read_text())["naturalness"]
delta = baseline - current
if delta > THRESHOLD:
print(f"REGRESSION: naturalness dropped {delta:.3f}")
else:
print(f"OK: {current:.3f} (baseline {baseline:.3f})")
This is a minimal example, not a production-ready snippet. It leaves out retry logic, error handling, and sub-score details. The important pattern is to score the batch, compare it with the baseline, and flag a delta beyond the threshold. Add handling and parallelism as your team's requirements expand.
When to escalate to a human listen
Automated scoring does not replace human judgment. It directs that attention. Certain patterns should lead to a human listen even when the aggregate score change is small:
- The overall score declines while the prosody sub-score remains steady. That combination can point to weaker pronunciation or intelligibility without a rhythm change, an issue automated metrics may capture less precisely in individual utterances.
- A checkpoint performs well on your evaluation set but will soon be tested with a new domain or speaking style. Automated MOS reflects the distribution used to calibrate the underlying model, so out-of-distribution audio deserves closer review before you rely on the score.
- Before a checkpoint enters any user-facing deployment, whatever its score. Automated scoring blocks obvious regressions before human review, but it is not a full replacement for that review.
A useful escalation rule is to request a focused human listen when the automated score remains acceptable but naturalness and pronunciation diverge unusually. Have someone spend 10 minutes on the utterance types showing the divergence, rather than repeating a full evaluation session.
Storing run history
A score kept only in a CI log loses value after the run ends. Save every run in persistent storage, such as a JSON file or SQLite database. At minimum, capture the checkpoint identifier, timestamp, aggregate naturalness, prosody sub-score, pronunciation sub-score, and utterance count. After a few weeks, you can chart the training path and see whether quality rose, leveled off, or oscillated. That pattern is more informative than one checkpoint score.
The trajectory reveals another regression pattern: gradual drift that stays under every single-checkpoint threshold. A model losing 0.03 at each checkpoint across five consecutive checkpoints has lost 0.15 overall, yet no individual run would flag it. A trend plot exposes that pattern.
One gap the loop will not catch
Evaluation sets are static. If the deployment context changes after launch, users may speak differently from the evaluation sentences or the TTS may serve a new domain. The daily loop can still report clean scores while users hear worse output. That mismatch is not a defect in automated evaluation. It shows why maintaining the evaluation set belongs in the process.
Revisit and update the evaluation sentences every few months, particularly after a significant change in product use. Add examples from the new domain and remove ones that no longer represent it. The loop is a starting point. Over time, teams often add speaker-level breakdowns, WER checks alongside MOS, and per-sentence floor thresholds for critical vocabulary. Begin simply, run it consistently, and expand from there.