How Speaker Diarization and Speaker Recognition Work

Speaker diarization is the part of transcription that answers "who spoke when". It takes a recording of several people, works out how many distinct voices there are, and marks which stretches of audio belong to each one, so a transcript reads as a conversation instead of a wall of text. This article explains the four stages behind it, how software goes one step further and recognises the same person across recordings, where the technology breaks, and what you can do on the recording side to get clean speaker labels.

Diarization versus speaker recognition

The two terms get mixed up, and the difference matters when you are judging a tool.

Diarization answers "which parts of this recording were spoken by the same voice". Its output is labels like Speaker 1, Speaker 2, Speaker 3. It does not know who those people are.

Speaker recognition (or speaker identification) answers "is this voice the one I have heard before". Given a stored example of a voice, it can say that Speaker 2 in today's meeting is the same person as the one you named last Tuesday. That is what lets a name stick across recordings.

Diarization on its own is enough for a one-off interview, which is what most people want when they go looking for interview transcription with speaker labels. Recognition is what makes speaker labels useful over weeks of recurring meetings, lectures, or services, because you stop relabelling the same people.

Stage 1: finding speech and cutting it into segments

The first job is to ignore everything that is not speech: silence, keyboard noise, a door, background music. A voice activity detector, a small model trained to answer "is someone talking right now", scans the audio in short windows and marks the speech regions.

The speech regions are then cut into segments. Older systems looked for points where the sound changed abruptly, on the assumption that a change of speaker produces a change in the acoustic signal. Newer systems simply slice speech into short overlapping windows, often around one to two seconds, and let the later stages decide which windows belong together. Short windows are better at catching quick turn-taking; longer windows produce a more reliable voice fingerprint. Every system is balancing those two.

Stage 2: turning each segment into a voice embedding

For each segment, a neural network produces an embedding: a list of a few hundred numbers that describes the voice, not the words. Two segments from the same person, even saying different things, end up with embeddings that are close together; two different people end up far apart. Pitch, timbre, resonance of the vocal tract, speaking rhythm, all of that is folded into those numbers in a way that is not human-readable but is remarkably consistent for a given voice.

The embedding model is trained on large collections of labelled speech, and its quality is the single biggest factor in how well diarization works. A good embedding model separates two adult voices of the same sex and similar age reliably on clean audio. Everything downstream depends on it.

Stage 3: clustering the embeddings into speakers

Now the software has hundreds or thousands of embeddings and needs to group them. That is clustering: put embeddings that are close together into the same group, and each group becomes one speaker. The hard question is how many groups there should be. If you tell the system "there are four people in this meeting" it can use that; if you do not, it has to decide the number itself by looking at how the embeddings spread out, and this is where over-splitting (one person becomes two) or merging (two people become one) tends to happen.

A common refinement is to run a second pass that re-examines segments near the boundaries between speakers, using the final clusters as a reference. This cleans up the moment where one person stops and the next starts, which in a first pass is often assigned a few hundred milliseconds late or early.

Stage 4: lining speakers up with words

Diarization and speech recognition are usually separate models. One produces "Speaker 2 from 14.3 s to 21.8 s"; the other produces words with their own timestamps. The final step aligns them: each word is given the speaker whose segment covers its timestamp. When a word sits exactly on a boundary, the system has to choose, and a sentence occasionally starts with a word attributed to the previous speaker. Word-level timestamps from the recogniser matter here; the more precise they are, the cleaner the handover in the transcript.

Recognising the same person across recordings

So far everything has been within one recording. To make names stick, the software keeps a compact voice profile for each named speaker: essentially an averaged embedding from the segments you confirmed belong to that person. In a new recording, each cluster's embedding is compared against the stored profiles. If a cluster is close enough to a stored profile, it inherits the name; if it is not close to anything, it becomes a new unnamed speaker.

This is why the first recording with a new group of people needs your input (you tell it who Speaker 1 and Speaker 2 are), and later recordings need less and less. It is also why the comparison needs a threshold: too loose and a colleague with a similar voice is mislabelled as you; too strict and you become "Speaker 3" every time you have a cold. Good systems let you correct a mislabel and use that correction to refine the profile.

Where this runs matters for privacy. A voice profile is biometric data. Systems that do all of this on the device keep those profiles on the device; cloud systems store them with the vendor. If you record clients, patients, or students, that difference belongs in your data policy.

Where diarization breaks

Diarization is good but not magic. The failure modes are predictable.

• Overlapping speech. When two people talk at once, a segment contains two voices and its embedding sits somewhere between them. Most systems attribute the overlap to one speaker or drop it. Cross-talk in a heated meeting is the hardest case there is.
• Similar voices. Siblings, colleagues of the same sex and age, or two people with the same regional accent produce embeddings that sit close together. Over a long recording the clusters usually separate, but short exchanges may be merged.
• Very short turns. "Yes", "right", "mm-hm" are under half a second, too short for a reliable embedding, and are often assigned to whoever was speaking around them.
• Poor audio. A single far-away microphone in a reverberant room blurs everyone's voice towards the room's own sound. A phone on a table in a large boardroom is the most common cause of bad speaker labels.
• Compressed call audio. Video-call audio is heavily processed: noise suppression, automatic gain, narrow bandwidth. Speakers are still separable, but the margin is smaller than with a local microphone, and some call apps mix everyone into one channel.
• Guessing the number of speakers. A silent participant who says one sentence in an hour may be merged into someone else; a speaker who moves closer to the microphone halfway through may be split into two.

How to get good speaker labels

Most of the improvement comes from the recording, not from settings.

1. Get the microphone close. One decent microphone within arm's reach of the speakers beats an expensive one across the room.
2. Record locally rather than through a call where you can. If you are recording a Zoom or Microsoft Teams call, capture the system audio directly; Record Zoom and Teams meetings on a Mac covers the options.
3. Reduce overlap where you have any control: a chair who lets people finish, a hand raise in a panel. The transcript improves as much as the meeting does.
4. Tell the software how many speakers there are, if it asks. It removes the hardest guess.
5. Name speakers early. In a system that recognises voices, a couple of confirmed recordings give it a solid profile; from then on you mostly correct exceptions.
6. Correct mislabels rather than ignoring them, if the tool learns from corrections.
7. Give it enough speech. Ten seconds of continuous talk from each person somewhere in the recording is usually enough to anchor a cluster; a round of introductions at the start does this naturally.

If speaker labels are consistently wrong in a recording, listen to the raw audio before blaming the model. Nine times out of ten the audio is the problem, and fixing the microphone position fixes the labels.

Where ThinkScribe fits

For the speech-recognition side of the pipeline, and how the two main on-device models compare, see Whisper vs Parakeet. For a worked example where speaker labels matter, the Q&A section of the sermon transcription guide walks through the options. The style, consent and anonymisation decisions that come with transcribing research interviews are covered in their own guide.

Frequently asked questions

What is speaker diarization?

Speaker diarization is the process of splitting a recording into segments by speaker and labelling which segments belong to the same voice, so a transcript can show who said what. It answers "who spoke when" without knowing the speakers' names. Combined with speaker recognition, which matches a voice against stored profiles, it can also attach names that persist across recordings.

How accurate is speaker diarization?

On clean audio with two to four distinct voices and little overlap, modern systems label the large majority of speech correctly and the errors cluster at turn boundaries and in very short interjections. Accuracy drops with more speakers, similar voices, overlapping speech, and a distant microphone. The recording setup matters more than the choice of software in most real situations.

Can diarization tell me who a speaker is by name?

Not by itself. Diarization produces anonymous labels. To get names, either you tell the software who each label is, or it compares the voices against profiles it has stored from previous recordings where you named people. Software that keeps those profiles can then label a known voice automatically in future recordings.

Does speaker diarization work on Zoom and phone calls?

Yes, with a smaller margin than a local microphone recording. Call audio is compressed and processed, and some apps mix all participants into one channel, so voices are a little harder to separate. Recording the call's system audio directly, rather than a room microphone next to a laptop speaker, gives the best result.