brain

Diarization

Properties7
statusseedling
typeconcept
created2026-09-23
updated2026-09-23
tagsstt, asr, audio, diarization, speech-processing
aliasesSpeaker diarization
public

3 min read

Diarization

Speaker diarization answers “who spoke when”: given an audio stream, it segments and labels each stretch of speech by speaker — without knowing the speakers in advance or who they are. It is the layer Speech to text inference deliberately omits: Parakeet TDT produces clean text but flattens every voice into one anonymous speaker, so a two-person meeting transcript becomes an unintelligible alternating monologue.

The standard metric is DER — Diarization Error Rate, the fraction of scored time misattributed (missed speech, false alarm, speaker confusion), overlap scored, no boundary collar. It says nothing about words: a model can have DER 0 and transcribe nothing (dedicated diarizers are usually paired with a separate ASR pass).

Nemotron-3-Diarization (announced 2026-09-23)

NVIDIA’s open-weight answer, announced the same day this note was planted. It is the current SOTA on Voice Arena’s Diarization-Bench: 14.72% DER, ranked #1 of 12 systems / 17 configurations, ~24% relative reduction over the next-ranked system (19.3%), evaluated on 139 English conversations (~22 h). Where the previous Sortformer baseline topped out at 4 speakers, Nemotron-3 expands to 8 speakers, 21 languages, both offline and online (streaming) modes, variable latency down to 0.32 s input-buffer, commercial and non-commercial use. 100M parameters — tiny by 2026 standards.

Throughput (DIHARD III, BF16, torch.compile, batch 32, RTX PRO 5000): 15,113x RTFx at the 30.4 s configuration (vs 2,619x baseline) while lowering DER 19.09% → 12.73%; 865x at 1.04 s (vs 136x) at DER 13.18%. At 1.04 s latency it cuts DER on all eight evaluation conditions, 9.0% (CALLHOME-Part2) to 65.2% (NOTSOFAR1 MHM) relative, ~40% average. The 0.32 s operating point is new; the baseline lacks a 0.64 s point entirely.

Architecture: audio in → arrival-ordered speaker-activity probabilities out, with AOSC and FIFO context supporting streaming (the blog does not expand the AOSC acronym). Output format is simply start end speaker_id lines. Accepts 16 kHz mono .wav/.flac/.opus/.mp3; requires Ampere+ GPU under NeMo/PyTorch — no CPU path documented. ASR pairing is via a separate integration guide, not built in.

Local relevance

Not a fit for the homelab today: NVIDIA-only toolchain while our STT lane is deliberately CPU-only (see Speech to text inference — GPU belongs to llama-swap). But diarization is where the STT stack’s next gap is:

  • transcribe.cpp (OSS) already ships dedicated diarizers on ggml — diar_streaming_sortformer_4spk-v2.1 (streaming Sortformer, DER/JER-verified), moss-transcribe-diarize (segment timestamps), plus combined diarize+transcribe models: Granite Speech 4.1 (word timestamps), multitalker-parakeet-streaming-0.6b-v1 (streaming + token timestamps).
  • parakeet.cpp (OSS) itself stays transcription-only — no diarization in its matrix.
  • The other usual suspect: pyannote.audio (the long-time open-source diarization standard, embedding + clustering pipeline). Not captured yet — red link.

The obvious homelab question is whether a meeting/brain-dump pipeline ever needs “who spoke when”. For single-speaker Telegram voice messages — today’s actual workload — the answer is no, and that is the correct default. It starts to matter the moment a transcript covers two people (meetings, interviews, Speech to text inference’s 5-minute brain-dump edge case where VAD chunking alone suffices).

Sources

Graph View

Backlinks