Speaker diarization on a Mac, locally: whisperX and pyannote

Speaker diarization on a Mac without a cloud service takes two models: a speech recognizer for the words and a diarization pipeline for who said them. The three working routes today are whisperX, which bundles both behind one command; pyannote.audio in Python, which you align yourself against whisper.cpp output; and FluidAudio, a Swift package that runs the same class of pipeline on Core ML. This guide covers all three with exact commands, realistic run times, accuracy advice and the errors people hit.

What you get and how it works

The output of diarization is a list of time ranges, each tagged with an anonymous label such as SPEAKER_00 or SPEAKER_01. Combined with a transcript it becomes a script:

[00:00:02 - 00:00:09] SPEAKER_00  Thanks for joining. Let's start with the budget.
[00:00:09 - 00:00:21] SPEAKER_01  Sure. The numbers moved since last week.

Every open pipeline here is built the same way, which the speaker diarization explainer covers in more depth:

1. Segmentation. A small neural network scans the audio in short windows and marks where speech is, where it changes, and where two people overlap. This doubles as voice activity detection.
2. Embeddings. Each speech region is turned into a fixed-length vector that captures the voice rather than the words. pyannote's current pipeline and FluidAudio both use WeSpeaker embeddings.
3. Clustering. The vectors are grouped; each group becomes one speaker label. The min_speakers and max_speakers hints constrain this step.

The speech recognizer is a separate model with its own timestamps; whisperX joins the two per word, and route B shows a small overlap script that does it per segment.

Before you start

• A Mac with Apple Silicon and 16 GB of memory is comfortable. 8 GB works with the smaller Whisper models.
• Python 3.10 or newer and ffmpeg (brew install ffmpeg). pyannote.audio 4 decodes audio through torchcodec, which needs ffmpeg installed.
• A free Hugging Face account and a read token from hf.co/settings/tokens.
• Acceptance of the gated model terms. For the current pipeline, open pyannote/speaker-diarization-community-1 while logged in and accept the conditions. The older pyannote/speaker-diarization-3.1 pipeline needs both pyannote/segmentation-3.0 and pyannote/speaker-diarization-3.1 accepted. Models download once and run offline afterwards.

Put the token in your shell so the commands below can reference it:

export HF_TOKEN=hf_xxxxxxxxxxxxxxxxxxxxx

Route A: whisperX, one command

whisperX wraps faster-whisper for recognition, a wav2vec2 alignment model for accurate word timings, and pyannote for diarization. As of writing it installs with pip or uv and defaults to pyannote/speaker-diarization-community-1 as the diarization model.

python3 -m venv ~/whisperx-env && source ~/whisperx-env/bin/activate
pip install whisperx

If you use uv, uvx whisperx runs it without a permanent install. Then:

whisperx meeting.wav --model large-v2 --device cpu --compute_type int8 \
  --language en --diarize --hf_token "$HF_TOKEN" \
  --min_speakers 2 --max_speakers 4 \
  --output_format srt --output_dir out

What each flag does, checked against the current command-line parser:

• --device cpu is the right choice on a Mac. The default is cuda when available and cpu otherwise; passing mps crashes, and the Apple GPU request was closed as not planned on the project tracker.
• --compute_type accepts default, float16, float32 and int8. float16 is a GPU type, so on CPU use int8 (fastest, slightly less accurate) or float32.
• --model defaults to small. large-v2 is the README's example; use small for a first run to confirm the setup.
• --language en skips detection and loads the right alignment model. Without it, whisperX detects the language from the first 30 seconds.
• --diarize turns on diarization; --hf_token is required for it. --min_speakers and --max_speakers bound the clustering. For a two-person call, set both to 2.
• --output_format is one of all, srt, vtt, txt, tsv, json, aud; the default is all. In SRT and TXT each line is prefixed [SPEAKER_00]:. The JSON has a speaker field on every segment and on every word.
• --batch_size defaults to 8; lower it if memory is tight.
• --highlight_words True underlines each word as it is spoken in the SRT, useful for checking alignment.

whisperX's README lists its limits plainly: overlapping speech is not handled well, diarization is "far from perfect", and words outside the alignment model's vocabulary (numbers, currency) get no timing.

Route B: pyannote.audio in Python, aligned to whisper.cpp

If you already transcribe with whisper.cpp as in run Whisper locally on a Mac, you can add diarization without changing that setup.

Install pyannote.audio:

python3 -m venv ~/pyannote-env && source ~/pyannote-env/bin/activate
pip install pyannote.audio

Run the pipeline and save the turns as a tab-separated file. This is the README's snippet with speaker bounds added and the output written to disk:

cat > diarize.py <<'PY'
import os, sys, torch
from pyannote.audio import Pipeline

pipeline = Pipeline.from_pretrained(
    "pyannote/speaker-diarization-community-1",
    token=os.environ["HF_TOKEN"])
pipeline.to(torch.device("cpu"))

output = pipeline(sys.argv[1], min_speakers=2, max_speakers=4)
with open(sys.argv[2], "w") as f:
    for turn, speaker in output.speaker_diarization:
        f.write(f"{turn.start:.3f}\t{turn.end:.3f}\t{speaker}\n")
PY
python diarize.py meeting.wav turns.tsv

num_speakers=2 replaces the min and max when you know the count exactly. Stereo or 44.1 kHz input is downmixed and resampled to 16 kHz by the pipeline, so no conversion step is needed here. pipeline.to(torch.device("mps")) is worth trying on Apple Silicon; if an operator error appears, go back to cpu.

Now produce whisper.cpp's JSON, which has millisecond offsets per segment:

ffmpeg -i meeting.m4a -ar 16000 -ac 1 -c:a pcm_s16le meeting.wav
whisper-cli -m ~/whisper-models/ggml-small.bin -f meeting.wav -l en -oj -of meeting

Assign each transcript segment to the speaker whose turn overlaps it most:

cat > merge.py <<'PY'
import json, sys

turns = [(float(s), float(e), spk) for s, e, spk in
         (line.rstrip("\n").split("\t") for line in open(sys.argv[2]))]
segments = json.load(open(sys.argv[1]))["transcription"]

for seg in segments:
    start = seg["offsets"]["from"] / 1000
    end = seg["offsets"]["to"] / 1000
    best, best_overlap = "UNKNOWN", 0.0
    for t_start, t_end, spk in turns:
        overlap = min(end, t_end) - max(start, t_start)
        if overlap > best_overlap:
            best, best_overlap = spk, overlap
    print(f"[{start:8.2f}] {best}: {seg['text'].strip()}")
PY
python merge.py meeting.json turns.tsv > meeting-speakers.txt

Largest-overlap assignment is what most tools do at segment level. Its weakness is a Whisper segment that spans a speaker change; -ml 40 (max segment length in characters) in whisper.cpp makes segments shorter and the assignment finer. whisperX avoids the problem by assigning per word.

Route C: FluidAudio on Core ML

FluidAudio is an Apache-licensed Swift package that runs speech-to-text, voice activity detection and speaker diarization as Core ML models on the Neural Engine. It targets macOS 14 and iOS 17 or later and requires Swift 6 tooling. The diarization pipeline follows the same design as pyannote's community-1 pipeline (segmentation, WeSpeaker embeddings, VBx clustering) converted to Core ML, and the models, around 100 MB, download from FluidInference/speaker-diarization-coreml into ~/Library/Application Support/FluidAudio/Models/ on first use. No Hugging Face token is involved.

It ships a command-line tool, macOS only, run from a checkout:

git clone https://github.com/FluidInference/FluidAudio.git
cd FluidAudio
swift run fluidaudiocli process meeting.wav --output results.json --threshold 0.6
swift run fluidaudiocli transcribe meeting.wav

process writes speaker segments to JSON; --threshold is the clustering threshold, with 0.6 used throughout the project's own examples. transcribe runs Parakeet, so a pair of commands gives you both halves, which you join with the same overlap logic as route B. In Swift the equivalent is a few lines: create an OfflineDiarizerManager, call prepareModels(), then process(audio:) and read segments with speakerId, startTimeSeconds and endTimeSeconds. The project's diarization documentation quotes a real-time factor of about 150x on an M2 MacBook Air and a diarization error rate of 17.7 percent on its benchmark set. Check the README before relying on a command, since the CLI syntax changes between releases.

Realistic run times on a Mac

Rough expectations for one hour of audio on an M-series Mac, as of writing. Diarization is timed separately from recognition because they are different models.

Step	Tool	Typical time for 1 h of audio
Recognition	whisperX, small, CPU int8	A few minutes
Recognition	whisperX, large-v2, CPU int8	Tens of minutes to over an hour
Recognition	whisper.cpp, small, Metal	A few minutes
Diarization	pyannote community-1, CPU	Minutes to tens of minutes, longer with many speakers
Diarization	FluidAudio, Neural Engine	Under a minute

Whisper on CPU is the slow part of routes A and B. If speed matters, combine whisper.cpp on Metal (route B) or FluidAudio's Parakeet with whichever diarizer you prefer.

Getting better labels

• Tell it how many speakers. min_speakers and max_speakers are the biggest lever. Two people on a phone call: set both to 2. A panel you are unsure about: a range such as 3 to 6 beats leaving it open.
• Microphone distance. A conference-room mic three meters from everyone yields embeddings that blur together. Per-person mics or a headset for the host changes the result more than any flag.
• Overlap. Crosstalk is where every pipeline loses most of its error rate. Expect the interrupter to be missed or mislabeled.
• Similar voices. Two speakers of the same sex, age and accent are the hard case; lowering the clustering threshold splits them at the cost of splitting single speakers elsewhere.
• Phone and video calls. Compressed, band-limited audio still works for two speakers. If five labels appear for a three-person call, one speaker was split; merge the labels with a search-and-replace.

Anonymous labels are not names

None of these tools know who SPEAKER_00 is. Turning labels into names requires a voice-enrollment step: store an embedding of a known person and compare new clusters against it. pyannote exposes embeddings, whisperX can return them with --speaker_embeddings, and FluidAudio's result includes a speaker database of vectors, but the matching and the store are yours to write. The speaker names guide shows how an app does that with a one-time naming step per person. For a two-voice recording the labels alone are usually enough, which is the whole difference between a wall of text and usable interview transcription with speaker labels.

Common errors and fixes

401 Client Error or Cannot access gated repo. You have not accepted the model conditions with the same account that owns the token, or the token has no read scope. Open the model page while logged in, accept, and create a fresh read token. For the 3.1 pipeline, both gated repositories need acceptance.

Could not load libtorchcodec or an audio decoding error in pyannote. pyannote.audio 4 decodes through torchcodec, which needs ffmpeg installed and a torchcodec build matching your torch version. brew install ffmpeg, then reinstall torch, torchaudio and torchcodec together. As a workaround, load the audio yourself and pass {"waveform": waveform, "sample_rate": sample_rate} to the pipeline instead of a path.

ValueError: unsupported device mps, or a crash after passing --device mps. faster-whisper, and therefore whisperX, run on CPU or CUDA only. Use --device cpu --compute_type int8.

float16 compute type error on CPU. Change --compute_type to int8 or float32.

whisper.cpp failed to read WAV file. whisper.cpp needs 16 kHz mono 16-bit PCM; run the ffmpeg conversion from route B. pyannote and whisperX accept other rates and resample internally.

No alignment model for your language. whisperX ships default wav2vec2 alignment models for a set of languages; for others pass --align_model with a Hugging Face model name, or use --no_align and accept segment-level timing.

Everything is one speaker. Usually a call where one side is much quieter, or a speaker-count hint that was too low. Normalize with ffmpeg's loudnorm filter and widen max_speakers.

If you'd rather not maintain this

Frequently asked questions

Does whisperX work on a Mac without a GPU?

Yes, on the CPU. Use --device cpu --compute_type int8. Recognition with the large models is slow this way, so start with small or medium, or use whisper.cpp on Metal for recognition and pyannote for diarization as in route B. Passing --device mps is not supported as of writing.

Why does pyannote need a Hugging Face token if it runs offline?

The pipeline weights are gated: you accept the model's terms on Hugging Face once, and the token proves that acceptance when the files are downloaded. After the download the pipeline runs with no network at all. The token is only needed again when a new model version is fetched.

Can I get real speaker names instead of SPEAKER_00?

Not from these tools by themselves. They cluster voices into anonymous labels. Naming requires enrolling each person's voice as an embedding and matching new recordings against it, which you would script on top of pyannote or FluidAudio, or get from an app that includes an enrollment step.

How accurate is local speaker diarization?

On clean recordings with two or three speakers and a known count, labels are right for most of the transcript and errors cluster at turn boundaries and in overlaps. On the public benchmarks pyannote reports, the community-1 pipeline's diarization error rate ranges from about 12 percent on meeting corpora to over 40 percent on the hardest sets, so results on far-field or crosstalk-heavy audio need checking.