Weekly checkpoint merges expose the limits of a pre-release voice check. If someone listens to a sample batch only a few days before shipping, a bug may be reported, or the samples may sound fine. That process works when iterations are infrequent. Once you merge training checkpoints weekly or run fine-tuning experiments daily, it breaks down. Regressions can build up between checks, leaving you to search several commits for the point where naturalness declined.
Podonos built its API for automated evaluation loops. This post shows how to connect it to GitHub Actions and stop a PR from merging when MOS falls.
Put voice quality in CI
Adding voice quality to CI follows the usual regression logic: earlier detection costs less. Finding a MOS decline at the checkpoint before release can mean examining dozens of model updates. Finding it on the introducing commit gives you a diff that can be understood in five minutes.
The common concern is that MOS is too noisy for a hard gate. That is reasonable when judging one sample. With a consistent test set, however, aggregate variance is small enough to reveal real regressions. In practice, 20 to 30 representative sentences can separate noise from an actual quality decline, as long as the threshold is set conservatively.
Set a MOS threshold
The 1-5 MOS scale defined by ITU-T P.800 provides useful reference points: below 3.0 is generally "fair" or lower, while scores above 4.0 are regarded as "good." A CI gate needs a sharper boundary, the score that distinguishes an acceptable model from one you would keep out of production.
Calibrate against your own baseline instead of relying on published figures. Score the current production model with the Podonos API on your test set and use that result as the floor. Place the gate 0.10 to 0.15 MOS points below it. This buffer should cover ordinary score variation while preserving sensitivity to real regressions.
Set the threshold for each voice or model variant rather than applying one global value. A multi-speaker model whose best speaker reaches 3.6 MOS should not be measured against a threshold designed for a single-speaker English system scoring 4.1. One shared threshold can falsely block the weaker speaker or overlook regressions in the stronger one.
GitHub Actions workflow
The example workflow starts after training finishes. Your checkpoint evaluator writes TTS outputs to a local artifact, and the workflow sends those files to the batch endpoint for scoring.
name: MOS quality gate
on:
workflow_run:
workflows: ["Train checkpoint"]
types: [completed]
jobs:
evaluate-mos:
runs-on: ubuntu-latest
if: ${{ github.event.workflow_run.conclusion == 'success' }}
steps:
- uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: "3.11"
- name: Install Podonos client
run: pip install podonos
- name: Download TTS outputs artifact
uses: actions/download-artifact@v4
with:
name: tts-outputs
path: ./eval_samples/
- name: Score samples and check threshold
env:
PODONOS_API_KEY: ${{ secrets.PODONOS_API_KEY }}
MOS_THRESHOLD: "3.75"
run: |
python scripts/evaluate_and_gate.py \
--audio-dir ./eval_samples/ \
--threshold $MOS_THRESHOLD \
--fail-on-regression
The evaluate_and_gate.py script sends the audio files in eval_samples/ to POST /v1/batch, polls for the results, and returns code 1 when mean MOS is under the threshold. That non-zero exit code fails the workflow step and prevents the PR from merging.
import os, sys, time, glob
import podonos
client = podonos.Client(api_key=os.environ["PODONOS_API_KEY"])
threshold = float(os.environ.get("MOS_THRESHOLD", "3.75"))
audio_files = glob.glob("./eval_samples/*.wav")
batch = client.evaluate_batch(audio_files)
while batch.status not in ("complete", "failed"):
time.sleep(5)
batch = client.get_batch(batch.id)
if batch.status == "failed":
print("Batch evaluation failed; check API logs")
sys.exit(1)
mean_mos = batch.mean_mos
print(f"Mean MOS: {mean_mos:.3f} (threshold: {threshold})")
if mean_mos < threshold:
print(f"FAIL: MOS {mean_mos:.3f} is below threshold {threshold}")
sys.exit(1)
print("PASS")
Separating noise from real regression
When a gate fails, first determine whether the score reflects a true regression or ordinary noise. Several checks can help.
Start by seeing whether the failure appears in a few samples or across the full test set. On Pro, the sub-score breakdown shows whether the change is mainly in prosody, naturalness, or pronunciation. A naturalness decline across every sentence often indicates a training artifact. A prosody decline limited to one phonetic or syntactic context suggests a training-data coverage gap.
Next, inspect the confidence intervals returned by the API. If the batch mean fell by 0.08 but the new and baseline intervals overlap substantially, the result remains statistically unclear. One workable rule is to require two consecutive runs below the threshold before confirming a regression, instead of blocking after one run.
Also maintain a baseline score file and refresh it after every merge to main. Compare against that rolling baseline instead of a fixed number. The threshold can then follow gradual gains without requiring a manual config change whenever a new production checkpoint ships.
Limits of CI evaluation
An automated MOS gate is an early warning, not a replacement for human evaluation. It detects regressions against a calibrated listener model, but not issues outside that model's distribution, including unusual accents, domain-specific pronunciation patterns, or expressive speech contexts that the calibration data does not represent well.
A "PASS" from the MOS gate is not final approval for a new voice. It shows that the model did not decline relative to the previous version, not that its quality has reached its potential. These claims differ. Confusing them can result in voices that are technically non-regressive but would still sound noticeably weaker in human evaluation.
The practical division is straightforward: let the CI gate block regressions automatically, then have people listen to every significant checkpoint before declaring a voice production-ready. The gate keeps listeners from reviewing every minor training run, while human evaluation covers situations automated scoring cannot. Both matter, and neither substitutes for the other.