NVIDIA Nemotron 3 Diarization: The 8-Speaker Open Model Explained
NVIDIA's Nemotron 3 Diarization tells up to 8 speakers apart in real time. How it works, its benchmark scores, the licence, and what it means for on-device apps.
On 23 September 2026 NVIDIA released Nemotron 3 Diarization, an open-weight model that answers "who spoke when" for up to eight speakers, either on a finished recording or live as the audio arrives. It replaces NVIDIA's Streaming Sortformer models, which topped out at four speakers, and on NVIDIA's own figures it cuts diarization errors by about 40% compared with them. It is also small, 100 million parameters, and it has already been converted to run on Apple's Neural Engine. This article explains what is new, how it works, and how to read its numbers.
If you are new to the topic, how speaker diarization works covers the basics.
What changed from Sortformer
| Streaming Sortformer v2 (2025) | Nemotron 3 Diarization (2026) | |
|---|---|---|
| Maximum speakers | 4 | 8 |
| Parameters | 117 million | 100 million |
| Languages trained | Mostly English | English, Mandarin, Hindi, Kannada, Telugu, Bengali |
| Training audio | About 2,400 h real + 5,150 h simulated | About 10,000 h real + 82,600 h simulated |
| Licence | CC-BY-4.0 | OpenMDW 1.1 |
Four speakers was the practical ceiling of the previous generation: at five or more, its error rate on the DIHARD III benchmark jumped from about 13% to over 42%. Eight covers most meetings, panels, classes and family recordings.
How it works
Classic diarization is a pipeline: find speech, cut it into short windows, turn each window into a voice fingerprint, then cluster the fingerprints. Nemotron 3 Diarization, like the Sortformer models it grew out of, is end-to-end: one neural network listens to the audio and directly outputs, for every 80 milliseconds, the probability that each of eight speakers is talking.
In more detail:
- Input. 16 kHz mono audio is turned into a mel spectrogram, a picture of the sound's energy across frequencies, every 10 ms. Eight of those frames are stacked into one 80 ms step.
- Encoder. A 31-layer Transformer with rotary position embeddings reads the sequence and builds a representation of who is speaking at each moment.
- Output. A final layer produces a grid of time steps by eight speaker slots, each cell a probability. Because each slot is independent, two slots can be active at once, so overlapping speech is handled naturally instead of being forced onto one speaker.
The trick that makes it work: sorting speakers by arrival
End-to-end diarization has a puzzle at its core. If a model has eight output slots, which slot should Alice go in? Nothing about Alice says "slot 3". Earlier systems handled this with permutation-invariant training, which tries every possible assignment and keeps the best one. That is expensive, and it gets much worse as the speaker count grows.
Sortformer, published by NVIDIA in 2024, uses a simpler rule called Sort Loss: slot 1 is whoever speaks first, slot 2 is whoever speaks second, and so on. The model learns to order speakers by their arrival time. That gives every speaker a consistent place without trying every permutation.
Streaming with a speaker cache
For live audio, the model cannot see the whole recording, so it needs to remember who it has already heard. The streaming version keeps two memories:
- a FIFO queue with the most recent stretch of audio features, for short-term context, and
- an Arrival-Order Speaker Cache: a compact store of the most characteristic moments of each speaker heard so far, kept in arrival order. When the queue overflows, the model chooses which frames best represent each speaker and keeps those.
With the cache, speaker 2 in minute 40 of a meeting is still speaker 2, even if they have not spoken since minute 3. The same weights work at several latencies: a 30.4-second buffer for offline use, or 1.04 s, 0.64 s and 0.32 s for live captioning.
The benchmark numbers
Diarization is scored with the diarization error rate (DER): the share of time that is labelled wrong, adding up speech that was missed, noise labelled as speech, and speech given to the wrong speaker. Lower is better. From the model card:
| Benchmark | Offline (30.4 s) | Live (1.04 s) | Live (0.32 s) |
|---|---|---|---|
| DIHARD III (varied real-world audio) | 12.73% | 13.18% | 13.55% |
| CALLHOME part 2 (phone calls) | 9.10% | 10.29% | |
| AMI (meetings, headset mix) | 9.25% | 9.48% | |
| AliMeeting (Mandarin meetings) | 6.40% | 6.59% |
Three things stand out. First, the live settings lose very little accuracy compared with offline, which is unusual. Second, on DIHARD III offline it scores 12.73% where the previous streaming Sortformer scored 19.09%. Third, NVIDIA reports first place on the independent VoiceArena diarization benchmark with 14.72% DER, ahead of 19.3% for the next system.
Two cautions. DER depends heavily on scoring rules (whether overlapping speech counts, whether there is a forgiveness margin at speaker changes), so numbers are comparable only within one benchmark and one set of rules. And a benchmark is not your recording: a noisy café, a speakerphone, or two siblings with similar voices can each behave very differently.
The licence question
There are two repositories and they are not the same:
- nvidia/Nemotron-3-Diarization is released under OpenMDW 1.1, an open licence that allows commercial use.
- nvidia/Nemotron-3-Diarization-preview is under NVIDIA's evaluation licence, which restricts use to internal evaluation on NVIDIA GPUs and forbids publishing results without permission.
If you are building on it, make sure you took the first one. NVIDIA's other new speech releases in 2026, including the Nemotron 3.5 streaming speech recogniser, use the same OpenMDW licence. Parakeet TDT and Canary remain CC-BY-4.0.
Running it on a Mac or iPhone
NVIDIA's own runtime is the NeMo toolkit on NVIDIA GPUs, where it runs thousands of times faster than real time. The more interesting news for personal devices is that FluidInference, which maintains the open-source FluidAudio library for Apple platforms, published Core ML conversions within a day of the release:
- a 190 MB half-precision build that scored 9.47% DER on the AMI meetings set in offline mode, around 900 times faster than real time on Apple silicon, and
- a 95 MB 8-bit build that runs entirely on the Neural Engine, at 9.63% to 9.76% DER depending on latency.
A 95 MB model that diarizes eight speakers live, on the Neural Engine, is small enough for a phone. That is the practical significance of this release: speaker labels for larger groups no longer need a server or a desktop GPU.
What it does not do
- It does not name anyone. It outputs slots in order of arrival, not identities. Recognising that slot 2 is Priya from last week's meeting needs a separate speaker-recognition step with stored voice profiles.
- It does not transcribe. You still need a speech recogniser, and a step that aligns words to speaker turns. NVIDIA suggests pairing it with Parakeet TDT or its Nemotron 3.5 speech model.
- Nine or more speakers will not fit in its eight slots.
- Language coverage is six languages in training. Voice characteristics carry across languages reasonably well, but test it on your own audio.
A notable trend in 2026 is models that do transcription and diarization in one pass, such as Microsoft's VibeVoice-ASR and OpenMOSS's MOSS-Transcribe-Diarize. NVIDIA is going the other way and keeping diarization a separate module. That separate-module approach is easier to run on small devices and to swap out when a better model arrives. For the wider landscape, see open speech recognition models in 2026.
Frequently asked questions
What is Nemotron 3 Diarization?
It is an open-weight speaker diarization model from NVIDIA, released on 23 September 2026. It detects which of up to eight speakers is talking at every 80 ms of audio, works both offline and live with latencies down to 0.32 seconds, and has 100 million parameters.
Is Nemotron 3 Diarization free for commercial use?
The main release, nvidia/Nemotron-3-Diarization, uses the OpenMDW 1.1 licence, which allows commercial use. A separate preview repository uses NVIDIA's evaluation licence, which does not. Check which one you downloaded.
How many speakers can it handle?
Up to eight. The Streaming Sortformer models it replaces were limited to four and degraded sharply beyond that.
How is it different from pyannote?
pyannote's pipelines combine a segmentation model, a speaker-embedding model and a clustering step. Nemotron 3 Diarization is a single end-to-end network that outputs speaker activity directly and streams in real time. Both are open. Which one is more accurate depends on the audio, and they are best compared on your own recordings.
Can it run on an iPhone?
Yes, through community Core ML conversions. An 8-bit build of about 95 MB runs fully on the Apple Neural Engine and supports live diarization.
Sources
- Nemotron 3 Diarization model card (Hugging Face)
- NVIDIA blog: Nemotron 3 Diarization
- Park et al., "Sortformer: Seamless Integration of Speaker Diarization and ASR by Bridging Timestamps and Tokens", arXiv:2409.06656
- Medennikov et al., "Streaming Sortformer: Speaker Cache-Based Online Speaker Diarization with Arrival-Time Ordering", arXiv:2507.18446
- Streaming Sortformer 4spk v2 model card
- Core ML conversion by FluidInference
- FluidAudio releases on GitHub
