Evaluating a speech-to-text vendor means testing four things on your own terms: accuracy on your audio, speed in your integration, compliance with your data obligations, and total cost at your volume. Public benchmarks and vendor claims help build a shortlist. The decision should rest on a private test set, a short pilot, and written answers to a fixed list of questions about data handling and contracts.
Key takeaways
- Screen on hard constraints first: languages, deployment region, data retention and training policy. They eliminate vendors faster than accuracy does.
- Compare accuracy only on your own audio, scored with one normaliser, including entity and identifier accuracy.
- Measure latency end to end from your own infrastructure, in the mode you will use: streaming, batch or turn-based.
- Model the full cost, including streaming versus batch rates, rounding, minimums and add-on features.
- Get data-protection answers in the contract, not only in marketing pages.
A staged process#
The order saves effort. There is no point measuring the accuracy of a vendor that cannot process data in the required region, or does not support one of your languages. Paper checks take hours. A proper accuracy comparison takes days. A pilot takes weeks.
Accuracy#
Build or reuse a private test set drawn from production traffic, with references transcribed from scratch. Run every shortlisted system on the same files, with the settings you would use in production, and score all outputs with the same normaliser. Report:
- Pooled WER overall and per stratum (channel, language, accent, noise).
- Recall and precision on the terms your application depends on, and exact-match rates for identifiers.
- Semantic WER if the transcript feeds an LLM or agent rather than a human reader.
- Confidence intervals, so a small gap is not mistaken for a real one.
Test each vendor's domain adaptation features, such as word lists, prompts or context, with the same input for all. A vendor that is behind out of the box may be ahead once given your terminology, or the reverse. Watch for over-biasing: check WER on ordinary words before and after adaptation.
Read a sample of errors by hand. Two systems with the same WER can fail very differently: one drops quiet words, another hallucinates text during silence, a third mangles numbers. The pattern tells you which failures your workflow can absorb.
Latency and throughput#
For streaming use, measure from your own servers, over the network path you will use in production. The numbers that matter are the delay from end of speech to final transcript, how quickly partial results appear, and how often partials are rewritten. For voice agents, add end-of-turn behaviour: how the system decides a speaker has finished, and whether it supports barge-in. See speech-to-text latency.
For batch use, measure turnaround time per file at realistic concurrency, not a single request. Ask about concurrency limits, rate limits and how the service behaves when they are hit: queuing, rejection, or degraded quality.
Data protection and security#
Voice recordings of identifiable people are personal data, and transcripts often contain special-category data such as health information. Ask each vendor in writing:
| Question | Why it matters |
|---|---|
| Where is audio processed and stored, including backups and logs? | Data residency and transfer rules |
| How long are audio and transcripts retained? | Minimisation; exposure in a breach |
| Is customer data used to train or improve models? | Purpose limitation; confidentiality |
| Is there a GDPR Article 28 DPA, and who are the sub-processors? | Legal basis for the processing |
| Which certifications apply, and to what scope? | Evidence of controls, e.g. ISO 27001 |
| Is on-premise or VPC deployment available? | Workloads that cannot leave your perimeter |
Check that the answers appear in the data processing agreement or the main contract. A policy page can change; a contract clause cannot change without your agreement. Where the use case falls under sector rules, such as healthcare or financial call recording, involve the relevant compliance team early. The EU AI Act may also apply, depending on what the transcripts are used for.
Cost#
List prices are rarely comparable directly. Build a cost model from your own traffic profile:
- Minutes per month, split into streaming and batch, since many vendors price them differently.
- Billing granularity: per second, per 15 seconds, per minute, and any minimum per request.
- Whether silence, multi-channel audio and failed requests are billed.
- Add-ons billed separately: diarization, redaction, language detection, domain adaptation.
- Committed-use discounts, overage rates and what happens when a commitment is not used.
Add engineering cost. An integration that needs a custom audio pipeline or retry logic has a cost that does not appear on the invoice. See speech-to-text pricing models.
The pilot#
A short production pilot with one or two finalists catches what offline tests miss: behaviour under real concurrency, audio from devices you did not sample, the quality of support when something breaks, and how users react to the transcripts. Agree success criteria before the pilot starts, using the metrics from the offline test, and a date to decide.
How Reson8 approaches this#
For readers screening Reson8 against the questions above: processing runs on EU GPUs, audio is not retained, customer data is not used for model training, a GDPR Article 28 DPA is available, and Resonate Labs is ISO 27001 certified (details at trust.reson8.dev). It covers nine European languages rather than a long tail, supports streaming, batch and a Turns endpoint for voice agents, and offers on-premise and VPC deployment on enterprise plans. Pricing is credit-based, with one minute of streaming costing two credits and one minute of batch one credit, and a free tier of 600 credits a month.
Common questions#
How many vendors should go through a full accuracy test?
Two to four. Fewer and you learn little; more and the transcription and integration effort grows without improving the decision.
Can I rely on a vendor's own benchmark numbers?
Use them to shortlist. Prefer numbers on public leaderboards with a stated method, and ask which test sets overlap the vendor's training data. Make the final call on your own audio.
What if the most accurate vendor fails the data checks?
Then it is not an option for that workload. Accuracy differences between strong systems are often a few points; a compliance failure is binary.
Should I plan for switching vendors?
Yes. Keep a thin abstraction over the API, store your test set and scoring scripts, and rerun the comparison yearly. Models improve quickly, and the ranking changes.
References#
- Regulation (EU) 2016/679 (General Data Protection Regulation), Article 28. EUR-Lex.
- Regulation (EU) 2024/1689 (Artificial Intelligence Act). EUR-Lex.
- ISO/IEC 27001:2022. Information security, cybersecurity and privacy protection: Information security management systems: Requirements. ISO.
- Srivastav, V. et al. Open ASR Leaderboard. Hugging Face.
- Bisani, M., & Ney, H. (2004). Bootstrap estimates for confidence intervals in ASR performance evaluation. ICASSP.