We chose a REST endpoint over a Python library for Podonos scoring. This post covers the reasoning behind that decision and shows how it fits into PyTorch training loops, Kaldi pipelines, and NeMo workflows.
Choosing against a library
A Python package seems convenient: direct function calls, offline operation, no network round-trip, and no API key handling. We prototyped one early, but problems appeared almost at once.
TTS training setups have strict dependency constraints. A NeMo team may already be pinning CUDA and PyTorch versions, along with APEX for mixed-precision training. Another package can introduce version conflicts. During development, one team told us that every new pip dependency required a week of internal review to assess the risk to its containerized training setup.
The larger concern is coupling scoring to the training environment. A scoring-model update would require a library release, followed by the same dependency review. With an API, those layers stay separate, so the model can improve without changes in your training repository.
POST /v1/evaluate
The main endpoint is one POST request. Upload a WAV or FLAC file as multipart data, and receive JSON containing the score breakdown. The request shape:
POST https://api.podonosvoice.com/v1/evaluate
Authorization: Bearer pdn_...
Content-Type: multipart/form-data
audio: [binary file data]
{
"scores": {
"mos": 3.87,
"naturalness": 3.91,
"prosody": 3.82,
"pronunciation": 3.94
},
"audio_duration_s": 4.2,
"model_version": "2025-08"
}
There is no setup to configure. Upload the file and receive its scores. The endpoint is stateless, with no session to maintain and no batch ID unless you choose the batch endpoint. Audio is not retained after scoring.
Adding it to PyTorch
A validation callback is the clearest insertion point in a PyTorch training loop. Create a fixed batch from the evaluation text set, save its waveforms in a temporary directory, score them, then send the results to your experiment tracker.
import requests, tempfile, soundfile as sf, os
def score_checkpoint_samples(samples, api_key):
url = "https://api.podonosvoice.com/v1/evaluate"
headers = {"Authorization": f"Bearer {api_key}"}
scores = []
with tempfile.TemporaryDirectory() as tmpdir:
for i, (waveform, sr) in enumerate(samples):
path = os.path.join(tmpdir, f"sample_{i}.wav")
sf.write(path, waveform.numpy(), sr)
with open(path, "rb") as f:
resp = requests.post(url, headers=headers, files={"audio": f})
scores.append(resp.json()["scores"]["naturalness"])
return sum(scores) / len(scores)
Your requirements.txt only needs `requests` and `soundfile` added if they are not already available, and both are common in TTS training environments.
Kaldi workflow integration
Kaldi workflows usually rely on shell orchestration. Add scoring to a run.sh or evaluate.sh script after synthesis inference finishes. Since shell can invoke curl directly, the REST API works here without any Python dependency:
#!/bin/bash
API_KEY="pdn_..."
AUDIO_DIR="exp/eval_output"
for wav in "$AUDIO_DIR"/*.wav; do
score=$(curl -s -X POST \
-H "Authorization: Bearer $API_KEY" \
-F "audio=@$wav" \
https://api.podonosvoice.com/v1/evaluate \
| python3 -c "import sys,json; print(json.load(sys.stdin)['scores']['naturalness'])")
echo "$wav $score"
done
Pipe the response into a log or calculate an average with a brief awk command. The Kaldi data directories and existing Kaldi scripts remain unchanged.
NeMo training callbacks
Through PyTorch Lightning, NeMo exposes callbacks. A custom callback can run quality scoring when each validation epoch ends:
from pytorch_lightning import Callback
import requests
class PodonosQualityCallback(Callback):
def __init__(self, eval_paths, api_key, threshold=3.5):
self.eval_paths = eval_paths
self.api_key = api_key
self.threshold = threshold
def on_validation_epoch_end(self, trainer, pl_module):
scores = []
for path in self.eval_paths:
with open(path, "rb") as f:
resp = requests.post(
"https://api.podonosvoice.com/v1/evaluate",
headers={"Authorization": f"Bearer {self.api_key}"},
files={"audio": f},
)
scores.append(resp.json()["scores"]["naturalness"])
avg = sum(scores) / len(scores)
trainer.logger.log_metrics({"pdn_naturalness": avg}, step=trainer.global_step)
if avg < self.threshold:
trainer.should_stop = True
The callback sends naturalness to your Lightning logger, such as wandb or tensorboard, or another configured logger. It can also halt training below a chosen floor. That stop is optional and may be too aggressive during active fine-tuning, where flagging could be preferable, but it shows the available control.
Where the API has limits
Network calls add latency. When teams score thousands of samples in one run, use the batch endpoint instead of separate POST requests. It accepts up to 500 audio URLs in JSON or multiple files as multipart data, returns a job ID, and provides results asynchronously. A 200-sample batch typically completes in under 90 seconds.
Synchronous scoring at every training step, instead of during epoch validation, is a real limitation for teams that need it. That pattern is uncommon in TTS training, where validation usually runs periodically rather than per batch. If your workflow depends on it, local inference through a library would fit better. This trade-off is part of the REST design, not something we want to hide.
The API suits most TTS team workflows, including checkpoint evaluation, nightly regression checks, and CI pipeline gates. We designed it around those patterns, where separation from the dependency tree is the better choice.