Mistral’s Voxtral Mini Ranks Second by Diarization Error Rate

Mistral AI’s four-billion-parameter Voxtral Mini Transcribe 3 ranked second by diarization error rate on a diarization and speech-recognition leaderboard, ahead of its larger sibling on the speaker-tracking measure.

Mini recorded a 40.4% diarization error rate (DER), placing it second among nine models. Voxtral Small Transcribe 3 scored 41.1% and ranked fourth. A lower DER indicates fewer diarization errors overall.

Diarization tracks who is speaking and when. The DER measure includes speech assigned to the wrong speaker, missed speech and speech wrongly flagged. It therefore captures more than whether a system labels each speaker correctly.

The evaluation covered 139 spontaneous English conversations with human-verified reference transcripts. Both Voxtral models correctly identified the number of speakers in 83.5% of the conversations. The test was limited to English conversations.

The ranking reversed on Artificial Analysis Word Error Rate (AA-WER), a separate measure of word accuracy. Mini scored 2.2% and placed fifth, while Small scored 2.1% and ranked third. Mini led on diarization, while Small had the better AA-WER result.

Overlapping speech remained a challenge: the models struggled when people spoke at the same time. The findings show different results across the two measures, with neither model leading both rankings.

Mistral introduced the Voxtral 3 suite on Oct. 20 alongside a voice-agent API. The Voxtral 3 voice-agent API launch covered the service and its pricing.