Speech recognition can run on the device that records the audio, or on servers reached over a network. On-device recognition works offline, keeps audio local and avoids a network round trip, but is limited to models small enough for the device's memory, compute and battery. Cloud recognition can run much larger models on data-centre GPUs, update them centrally, and serve any client, at the cost of a network dependency and sending audio to someone else's infrastructure. Many real products are hybrids, and a third option, running the server-side model inside your own perimeter, addresses the privacy concern without shrinking the model.
Key takeaways
- The core trade-off is model size. Devices run models from millions to a few hundred million parameters; servers can run far larger ones at high utilisation.
- On-device wins on offline operation and on removing network delay. Cloud wins on accuracy for open vocabulary, long audio, many languages and domain terms.
- Privacy is not only a device question. On-premise or VPC deployment keeps audio inside your organisation while using server-class models.
- Hybrid designs are the norm for assistants: a local wake word and commands, cloud transcription for everything else.
- Cost moves, it does not vanish. On-device shifts compute to users' hardware and development effort to your team.
Where the model can run#
It is more accurate to think of three placements than two, distinguished by who controls the hardware and where the audio travels.
- On device: phones, laptops, cars, hearing aids, smart speakers, industrial handhelds.
- Your perimeter: servers in your own data centre or a virtual private cloud you control. See on-premise and VPC deployment.
- Provider cloud: a managed API. The provider runs, scales and updates the model. See integrating a speech-to-text API.
The trade-offs#
| On device | Cloud (provider or own servers) | |
|---|---|---|
| Model size | Bounded by device memory, compute and battery | Bounded by GPU memory and cost; can be much larger |
| Accuracy on open vocabulary | Lower, especially rare words and long-tail domains | Higher, with room for domain adaptation |
| Network dependency | None | Required |
| Latency | No network hop; compute may be slower | Network round trip; fast compute |
| Languages | Usually a few per downloaded model | Many, switchable per request |
| Updates | Shipped to every device | Central, immediate |
| Cost | User hardware, engineering for many targets | Per-use fees or GPU operating cost |
| Data control | Audio can stay on the device | Depends on provider terms, region and retention |
On-device recognition#
On-device recognition has been practical on phones for about a decade. McGraw et al. (2016) described an offline recogniser for mobile devices, and He et al. (2019) described a streaming end-to-end RNN-T recogniser compact and fast enough to run on a phone while remaining competitive in accuracy with a conventional server-side system. Operating systems now ship local dictation, and open-source runtimes such as whisper.cpp run quantised Whisper models on laptops and phones.
The engineering challenge is fitting a good model into tight limits. Typical techniques are smaller architectures, quantisation to 8 or 4 bits, distillation from a larger model, and use of dedicated neural accelerators where the hardware has them. Each device family needs its own build, testing and performance tuning.
Where it fits best#
- Wake words and commands, which must always be available and cannot send every sound to a server. See keyword spotting and wake words.
- Offline environments: cars in tunnels, aircraft, field work, sites with no connectivity. See automotive voice recognition.
- Short, latency-sensitive interactions on devices with capable accelerators.
- Personal dictation where the user's audio should never leave the device.
Cloud recognition#
Server-side models can be an order of magnitude or more larger than what fits on a phone, run on data-centre GPUs, and serve many users from the same hardware through batching. See GPU inference for speech. They tend to perform better on the hard cases: long-form audio, meetings, telephony, many languages, code-switching, and specialist vocabulary through context biasing. Updates reach every user at once.
The network costs time: a round trip from a European client to a European data centre is typically tens of milliseconds, more over mobile networks. For streaming, that delay is added once per result rather than once per word, and it is usually small next to model and endpointing delay. See speech-to-text latency.
The larger question is data. Sending audio to a provider means it is processed on their infrastructure, in a region they operate, under their retention policy. Under GDPR the provider is a processor and needs a data processing agreement, and the processing region matters for data residency.
Hybrid designs#
Most voice assistants combine both. A local detector listens for the wake word; a local model may handle a fixed command set such as "next track" offline; open-ended requests go to a cloud recogniser. Other hybrids run a small local model for instant partial results and a server model for the final transcript, or fall back to on-device recognition when the network is unavailable.
How Reson8 approaches this#
Reson8 runs Resonant-1 as a server-side model on EU GPU infrastructure, with zero audio retention and no training on customer data. For organisations that need audio to stay inside their own perimeter, on-premise and VPC deployment is available on enterprise plans, which gives server-class accuracy without sending audio to Reson8's infrastructure. Security documentation is at the trust centre.
Common questions#
Is on-device recognition always more private?
It avoids sending audio to a server, which removes a large class of risk. But transcripts are often synced or sent elsewhere afterwards, and a cloud or self-hosted service with zero retention and clear contracts can also meet strict requirements.
Is on-device faster?
It removes the network hop. Whether the total is faster depends on the device's compute; a large model on a slow processor can be slower than a server round trip.
Can on-device models use custom vocabulary?
Some support word lists or biasing. Large domain vocabularies and long context are much easier on servers, where memory is less constrained.
References#
- He, Y. et al. (2019). Streaming End-to-end Speech Recognition for Mobile Devices. ICASSP.
- McGraw, I. et al. (2016). Personalized Speech Recognition on Mobile Devices. ICASSP.
- Warden, P., & Situnayake, D. (2019). TinyML: Machine Learning with TensorFlow Lite on Arduino and Ultra-Low-Power Microcontrollers. O'Reilly.
- whisper.cpp. GitHub.
- Regulation (EU) 2016/679 (General Data Protection Regulation), Article 28. EUR-Lex.