Why Whisper hallucinates and repeats itself, and how to stop it
Whisper hallucination and repetition loops explained: why silence becomes 'Thank you for watching' and the whisper.cpp, openai-whisper and faster-whisper fixes.
Whisper hallucinates because its decoder is a language model that must produce text for every window of audio, and when the window holds silence, music or noise it produces the most likely text from its training data instead of nothing. Repetition loops come from the same decoder feeding its own previous output back in as context, so one bad sentence becomes the prompt for the next. Both are fixable: remove silence before decoding, stop the model from conditioning on its previous text, and let the built-in thresholds reject bad windows. This article explains the mechanism, then the settings, with verified flag names for whisper.cpp, openai-whisper and faster-whisper.
What it looks like
Three patterns account for nearly all reports:
- Phantom text over silence. A pause, an intro jingle or a muted microphone yields "Thank you for watching", "Subtitles by the Amara.org community" or a plausible sentence in another language. In one published analysis of Whisper on non-speech audio, "thank you", "thanks for watching" and "thank you for watching" together made up more than a third of the hallucinated phrases.
- A sentence that repeats. The same clause appears ten or fifty times with shifting timestamps until the next window breaks the pattern or the file ends.
- Drift. After a bad window the transcript stays subtly wrong for a while, with fragments, the wrong language or invented names, then recovers.
The mechanism
Whisper is an encoder-decoder transformer. The encoder turns 30 seconds of audio into features; the decoder predicts the next token given those features and every token it has already written. That is a text language model with an audio input attached, and three consequences follow.
It has to write something. The decoder emits tokens until it produces an end marker. Given silence, the audio features carry no information and the text prior decides alone. That prior was learned from around 680,000 hours of web audio paired with subtitles, which routinely contain sign-offs and credits placed over music or silence. The model learned that silence is where those phrases go.
Its own output is its prompt. By default, implementations carry the tokens from the previous window into the next one as a prefix, which keeps names and punctuation consistent across a long file. When the previous window was a hallucination or a loop, the prefix is garbage, and the most likely continuation of garbage is more of the same. This is why a loop, once started, can run for minutes.
It has no hard stop. A special no-speech token gives a decent silence estimate, and the implementations use it, but only as a check after decoding, not as a gate before it.
Every fix below strengthens the audio signal (send no silence), cuts the feedback path (no conditioning on previous text), or catches the failure afterwards (thresholds and fallbacks).
Fix 1: never feed it silence
The most effective change. Voice activity detection (VAD) passes only speech regions to Whisper, so the model never sees a window with nothing to anchor on, and the file gets shorter and faster.
whisper.cpp has a --vad option backed by a Silero VAD model in ggml format. As of writing the repository's download script offers silero-v5.1.2 and silero-v6.2.0, and the file is under 1 MB:
curl -L -o ~/whisper-models/ggml-silero-v6.2.0.bin \
https://huggingface.co/ggml-org/whisper-vad/resolve/main/ggml-silero-v6.2.0.bin
whisper-cli -m ~/whisper-models/ggml-small.bin -f talk.wav -l en \
--vad -vm ~/whisper-models/ggml-silero-v6.2.0.bin -otxt
The tuning flags and their defaults from the current CLI: --vad-threshold 0.5 (speech probability), --vad-min-speech-duration-ms 250, --vad-min-silence-duration-ms 100, --vad-max-speech-duration-s (unbounded by default; set it to split long stretches), --vad-speech-pad-ms 30 and --vad-samples-overlap 0.1. If the VAD clips the first syllable of sentences, raise the padding to 100 or 200 ms.
faster-whisper bundles Silero VAD and turns it on with vad_filter=True. Its VadOptions defaults are threshold=0.5, min_speech_duration_ms=0, max_speech_duration_s=inf, min_silence_duration_ms=2000 and speech_pad_ms=400:
python3 -c '
from faster_whisper import WhisperModel
model = WhisperModel("small", device="cpu", compute_type="int8")
segments, info = model.transcribe("talk.wav", language="en",
vad_filter=True, vad_parameters=dict(min_silence_duration_ms=500))
for s in segments:
print(f"[{s.start:7.2f} -> {s.end:7.2f}] {s.text}")
'
openai-whisper has no VAD. Trim silence with ffmpeg first. The silenceremove filter with a negative stop_periods removes silence anywhere in the file; this keeps any pause under one second and cuts longer ones, using a -40 dB floor that suits speech recordings:
ffmpeg -i talk.m4a -af "silenceremove=stop_periods=-1:stop_duration=1:stop_threshold=-40dB" \
-ar 16000 -ac 1 -c:a pcm_s16le talk-trimmed.wav
Output timestamps then refer to the trimmed file, so use this route for text rather than subtitles; whisper.cpp and faster-whisper's VAD keep the original timeline.
Fix 2: set the language
Auto-detection looks at the first 30 seconds. If those are music, silence or a different language, the whole file is decoded with the wrong language token, and a wrong-language decoder hallucinates freely. Pass -l en in whisper.cpp, --language en in openai-whisper, language="en" in faster-whisper. The English-only .en models cannot pick the wrong language at all.
Fix 3: cut the feedback loop
This is the fix for repetition specifically. Each implementation names it differently:
| Implementation | Setting | Default |
|---|---|---|
| openai-whisper | --condition_on_previous_text False (CLI) or condition_on_previous_text=False (Python) |
True |
| faster-whisper | condition_on_previous_text=False |
True |
| whisperX | --condition_on_previous_text |
False |
| whisper.cpp | -mc 0 (--max-context 0, no previous-text tokens carried) |
-1, meaning the model's full limit |
whisper.cpp's CLI has no separate no-context switch as of writing; --max-context sets how many tokens of previous text the decoder may see, and zero disables it. Both whisper.cpp and faster-whisper (prompt_reset_on_temperature=0.5) also drop the previous text automatically once the temperature fallback reaches 0.5. Turning conditioning off costs a little consistency in names and punctuation; on recordings with long pauses, the trade is worth it.
Fix 4: temperature fallback and the three thresholds
All three implementations decode a window, then judge the result, and if it fails they decode again at a higher temperature. The judges are the same, with the same defaults, in openai-whisper, faster-whisper (log_prob_threshold is the name there) and whisper.cpp:
| Check | Meaning | Default |
|---|---|---|
compression_ratio_threshold (whisper.cpp: -et, entropy) |
Output that gzips too well is repetitive; treat as failed | 2.4 |
logprob_threshold (whisper.cpp: -lpt) |
Average token log-probability below this is a failed decode | -1.0 |
no_speech_threshold (whisper.cpp: -nth) |
If the no-speech probability is above this and the decode also failed the logprob check, the window is silence: emit nothing | 0.6 |
The temperature schedule is 0.0, 0.2, 0.4, 0.6, 0.8, 1.0 in the Python implementations; whisper.cpp uses -tp 0 and -tpi 0.2 for the same ladder, and -nf disables fallback. A stuck loop at temperature 0 is exactly what the compression check catches, and the retry at 0.2 or 0.4 usually breaks it.
When hallucinations survive the defaults:
- Lower
no_speech_thresholdto 0.4 or 0.5 so borderline silence is dropped rather than decoded. - Raise
logprob_thresholdtoward -0.8 to reject low-confidence windows earlier. - In faster-whisper,
no_repeat_ngram_size=3forbids any three-token sequence from recurring within a window,repetition_penalty=1.1discourages reuse more softly, andhallucination_silence_threshold=2.0(seconds, requiresword_timestamps=True) skips silent stretches where a hallucination is suspected. openai-whisper has the samehallucination_silence_threshold. - In whisper.cpp,
-snssuppresses non-speech tokens such as bracketed sound descriptions.
Fix 5: cap the segment length
A loop needs room to run. whisper.cpp's -ml 60 limits a segment to 60 characters and -sow splits on word boundaries; faster-whisper's max_new_tokens caps tokens per window. Neither stops a loop from starting, but a capped loop yields two repeated lines instead of two hundred.
Fix 6: give it a prompt
An initial prompt sets the style and vocabulary the decoder expects. A correctly punctuated sentence containing the names and jargon from the recording biases spelling and gives the first window a sane prefix instead of an empty one. Use --prompt in whisper.cpp, --initial_prompt in openai-whisper, initial_prompt in faster-whisper, which also has hotwords for terms alone. Keep prompts short; whisper.cpp caps them at half the text context.
Fix 7: pick the right model
Larger models hallucinate less on borderline audio, and large-v3-turbo keeps most of large's quality at a fraction of the time. For English, the .en variants are more stable than multilingual ones of the same size. tiny and base are the most loop-prone; if a file misbehaves on base, try small before touching thresholds.
Fix 8: chunk long files at silences
A two-hour file is 240 windows in sequence, each a chance for one bad window to poison the next. Cutting the file at natural pauses contains the damage and lets you re-run only the piece that failed. --vad-max-speech-duration-s 300 in whisper.cpp splits speech regions longer than five minutes at a silence; max_speech_duration_s does the same in faster-whisper. Without VAD, ffmpeg -f segment -segment_time 600 cuts fixed ten-minute pieces, cruder but workable.
Detecting loops after the fact
Even with the safeguards on, check the output. A repeated n-gram count catches loops in any transcript in a second:
cat > find_loops.py <<'PY'
import re, sys
from collections import Counter
words = re.findall(r"\w+", open(sys.argv[1]).read().lower())
n = int(sys.argv[2]) if len(sys.argv) > 2 else 6
grams = Counter(tuple(words[i:i + n]) for i in range(len(words) - n + 1))
for gram, count in grams.most_common(10):
if count >= 3:
print(f"{count:4d}x {' '.join(gram)}")
PY
python3 find_loops.py talk.txt 6
A six-word phrase that appears three or more times is almost always a loop, unless the recording is a chant. Grep for the usual suspects as well: grep -in "thank you for watching\|subtitles by\|amara" talk.txt.
Why Parakeet loops less, and when it still does
Transducer and CTC models such as NVIDIA's Parakeet decode differently. The decoder advances frame by frame with the audio and emits a token or a blank at each step, so the amount of text is bounded by the number of audio frames. Given silence, the natural output is a run of blanks, and "Thank you for watching" over a pause largely disappears. The Whisper vs Parakeet comparison covers the other differences.
They are not immune. Very long inputs push the encoder's attention past what it was trained on, and the symptom is a stretch that repeats or collapses into fragments. The Parakeet TDT model card states that the model handles up to 24 minutes in one pass with full attention and up to about three hours with local attention, on a large GPU; on a Mac the memory limit arrives sooner. The remedy is Fix 8 again: chunk at silences into pieces of a few minutes, transcribe each, and concatenate with offset timestamps. The same model can look flawless in a tool that chunks internally and stutter in a script that feeds it an hour in one call.
Do not summarize a transcript you have not checked
A hallucinated transcript handed to a language model for summarization produces a confident summary of things nobody said. The summarizer never hears the audio; "Thank you for watching" repeated forty times becomes a meeting that ended with prolonged thanks, and an invented sentence about a deadline becomes an action item. Run the loop detector and skim around long pauses before the transcript goes further, whether the next step is a local LLM summary, a cloud one, or a search index. Text is trusted downstream in a way audio never is, so the gate belongs at the transcript.
Where ThinkScribe fits
Frequently asked questions
Why does Whisper output "Thank you for watching" on silence?
Because its decoder is a text language model trained on subtitled web video, where sign-off phrases sit over silence or music. When a 30-second window contains no speech, the decoder has nothing to go on and emits the text it learned belongs there. Removing silence with a VAD before decoding is the fix.
Why does Whisper repeat the same sentence over and over?
The decoder conditions each window on the text it produced for the previous window. Once a window goes wrong, the bad text becomes the prompt for the next one and the model continues the pattern. Disable conditioning on previous text, keep the temperature fallback on, and cap segment length so a loop that does start stays short.
Does whisper.cpp have a fix for hallucinations?
Yes. Use --vad with a Silero VAD model to skip silence, -l to fix the language, -mc 0 to stop carrying previous text, and the -et, -lpt and -nth thresholds with temperature fallback to reject bad windows. -ml with -sow caps how far a loop can run.
Does Parakeet hallucinate?
Much less on silence, because a transducer decoder is tied to audio frames and emits blanks when there is nothing to transcribe. It can still repeat or degrade on inputs longer than the model was trained for, which is why long recordings should be chunked at pauses before transcription.
