Ambient clinical documentation records a consultation, transcribes the conversation with speaker labels, and uses a language model to draft a structured clinical note that the clinician reviews and signs. The aim is to give clinicians back time spent typing. The risks are specific: words misheard, statements attributed to the wrong person, and findings the model added or left out.
Key takeaways
- An ambient scribe is a pipeline: capture, speech recognition with diarisation, note generation, clinician review, and filing in the health record.
- Each stage adds its own error type. ASR mishears, diarisation misattributes, the language model omits or invents. Review has to catch all three.
- Conversation is harder than dictation: far-field audio, overlapping speakers, lay vocabulary and hypotheticals ("if the pain gets worse...").
- Recordings contain health data, a special category under GDPR Article 9. Patients need clear information, and a data protection impact assessment is usually required.
- The transcript is the foundation. A note generated from a transcript that missed "not" is confidently wrong.
How an ambient scribe works#
Capture is usually a smartphone app, a desk microphone or a device in the room. For video consultations, the scribe can take the call audio directly, often one channel per participant.
Transcription runs either streaming, so the draft is ready seconds after the patient leaves, or in batch after the recording ends. Diarisation separates clinician from patient and any relatives or interpreters.
Note generation prompts a language model with the transcript and a template: SOAP, a specialty-specific format, or the hospital's own letter structure. Some products also suggest diagnosis or procedure codes, which raises separate regulatory questions (see below).
Review is not optional. The clinician remains responsible for the content of the record.
Why consultations are harder than dictation#
Dictation is one trained speaker, close to a microphone, using clinical vocabulary deliberately. A consultation is none of these.
- Far-field audio. A phone on a desk picks up keyboard noise, a paper couch cover, the corridor. See transcribing noisy audio.
- Several speakers who overlap. Patients interrupt, relatives answer for them, a child talks in the background.
- Lay vocabulary. The patient says my water tablets; the note should say "furosemide 40 mg" only if the clinician confirmed the drug. A model that fills in the likely drug is guessing.
- Hypotheticals and negations. If you get chest pain, call 112 is safety-netting advice, not a symptom. No, I haven't had any blood in my stool must survive as a negative.
- Multilingual consultations. An interpreter, or a family member, switches between languages. See code-switching.
A taxonomy of errors#
| Stage | Error | Example |
|---|---|---|
| ASR | Substitution of a clinical term | "ramipril" transcribed as "ram a pill"; the note omits the drug |
| ASR | Dropped negation | "not short of breath" becomes "short of breath" |
| Diarisation | Misattribution | A daughter's "I've got diabetes too" recorded as the patient's history |
| Generation | Omission | A side effect the patient mentioned in passing never reaches the note |
| Generation | Hallucination | "Examination unremarkable" when no examination was discussed |
| Generation | Over-interpretation | "Chest pain" summarised as "angina" |
Evaluation should score each stage. For the transcript, use WER plus medical term recall (see medical speech recognition) and a diarisation metric. For the note, clinicians compare it against the recording and count omissions, unsupported statements and misattributions. Scores from a general summarisation benchmark tell you little about any of these.
What the evidence says#
The motivation is documentation burden. A time-and-motion study by Sinsky and colleagues (2016) observed US ambulatory physicians spending about two hours on EHR and desk work for every hour of direct patient time. Ambient scribes target that ratio.
Early large deployments have reported positive clinician experience. Tierney and colleagues (2024) described the rollout of an ambient AI scribe across The Permanente Medical Group, with broad uptake among physicians, reported time savings, and clinician feedback that patients received more attention during visits. They also reported inaccuracies in some notes, reinforcing the need for review. Evidence on patient outcomes and on error rates in signed notes is still limited, and results from one health system and language do not transfer automatically to another.
Consent, privacy and regulation#
Data protection#
A consultation recording contains data concerning health, a special category under GDPR Article 9. Care providers typically rely on Article 9(2)(h) for processing necessary for providing health care, rather than on consent, but must still inform patients under Article 13 in plain language: that the consultation is recorded, why, who processes it and how long it is kept. Many providers also ask patients whether they are comfortable with recording, and let them decline. Large-scale processing of special category data generally requires a data protection impact assessment under Article 35(3)(b).
Storage limitation (Article 5(1)(e)) argues for deleting audio once the note is signed. The transcript and the note are themselves health data and follow the retention rules of the medical record. The scribe vendor, and any speech recognition provider behind it, act as processors and need Article 28 agreements.
Medical devices and AI Act#
Whether an ambient scribe is a medical device under Regulation (EU) 2017/745 depends on its intended purpose. A tool that drafts documentation for clinician review is generally treated as outside the definition; one that suggests diagnoses, flags risks or recommends treatment may fall inside it, following the qualification logic in MDCG 2019-11. If it is a device requiring third-party conformity assessment, the EU AI Act classes it as high-risk. See the EU AI Act and speech recognition.
How Reson8 approaches this#
Reson8 provides the speech recognition layer, not the note generator. Resonant-1 runs in real-time streaming or batch mode, and text-only domain adaptation lets a scribe vendor supply specialty vocabulary, supporting 50,000+ medical terms without latency impact. Audio is processed on EU GPUs with zero retention and no training on customer data, under a GDPR Article 28 DPA. The founders previously built large-scale medical transcription platforms used in daily clinical workflows.
Common questions#
Do patients have to consent to recording?
GDPR does not always require consent as the legal basis; health care providers often rely on Article 9(2)(h). Patients must still be informed, and national law, professional rules and local policy may require asking permission. Check with your data protection officer.
Should the audio be kept?
Keeping it helps audit disputed notes; deleting it reduces risk and follows storage limitation. Many deployments keep audio only until the note is signed, and retain the transcript with the record.
Can an ambient scribe work in Dutch or German consultations?
Yes, if both the speech model and the note generator handle the language well, including the mix of local terms, Latin and English that clinicians use. Test on real consultations in the target language, not translated English ones.
Is streaming or batch better?
Streaming lets the draft appear within seconds of the consultation ending. Batch can use the whole recording as context and costs less. Both are viable; the choice depends on whether clinicians review immediately or at the end of a session.
References#
- Sinsky, C. et al. (2016). Allocation of Physician Time in Ambulatory Practice: A Time and Motion Study in 4 Specialties. Annals of Internal Medicine, 165(11).
- Tierney, A. A. et al. (2024). Ambient Artificial Intelligence Scribes to Alleviate the Burden of Clinical Documentation. NEJM Catalyst Innovations in Care Delivery, 5(3).
- Hodgson, T., & Coiera, E. (2016). Risks and benefits of speech recognition for clinical documentation: a systematic review. Journal of the American Medical Informatics Association, 23(e1).
- Regulation (EU) 2016/679 (General Data Protection Regulation). EUR-Lex.
- Regulation (EU) 2017/745 on medical devices. EUR-Lex.
- Medical Device Coordination Group (2019). MDCG 2019-11: Guidance on Qualification and Classification of Software. European Commission.
- Regulation (EU) 2024/1689 (Artificial Intelligence Act). EUR-Lex.