Benchmarks — Mumbli STT Providers

Current latency, quality, and cost benchmarks for Mumbli’s speech-to-text providers: ElevenLabs, Groq, and Deepgram.

Benchmarks

Current provider results for Mumbli’s speech-to-text engines.

Last updated: June 3, 2026

TL;DR

Need	Pick	Why
Best quality	Standard: ElevenLabs Scribe + GPT-5.4 Nano	ElevenLabs had the most audio-judge wins: 16/50, versus 4 for Deepgram and 3 for Groq.
Fastest	Fast: Groq Whisper + Groq Llama 3.1 8B	Groq had the lowest median STT latency: 1,048ms, with the best p95 at 3,037ms.
Cheapest STT	Groq Whisper large-v3-turbo	Current listed STT price is $0.04/hour transcribed, lower than ElevenLabs Scribe at$0.22/hour and Deepgram Nova-3 at roughly $0.29/hour.
Best middle option	Deepgram: Nova-3 + GPT-5.4 Nano	Reliable 50/50 success, often faster than ElevenLabs, but weaker than ElevenLabs on judged quality.

Pricing was checked on June 3, 2026 from the official Groq pricing, ElevenLabs API pricing, and Deepgram pricing pages. Provider pricing changes; use the latency and quality results below as the durable part of the benchmark.

Current STT benchmark

We benchmarked the latest 50 saved Mumbli recordings: 896 seconds of real dictation, ranging from 0.9s to 90.7s per clip.

Provider	Model	Success	Median STT latency	p95 latency	Judge wins
Groq Whisper	whisper-large-v3-turbo	50/50	1,048ms	3,037ms	3
Deepgram Nova-3	nova-3	50/50	1,928ms	6,831ms	4
ElevenLabs Scribe	scribe_v1	50/50	2,538ms	8,570ms	16
Interfaze Run Task	interfaze-beta	50/50	22,206ms	54,628ms	2
Interfaze STT	interfaze-beta	50/50	22,371ms	36,445ms	3
Tie / skipped	—	—	—	—	19 ties, 3 skipped

Conclusion: ElevenLabs remains the best default transcription engine for quality. Deepgram Nova-3 is reliable and often faster than ElevenLabs, especially on longer clips, but it did not beat ElevenLabs on judged quality and sometimes returned empty text on very short clips. Groq is still the fastest option by far and remains the right engine when low latency matters more than exact wording. Interfaze worked reliably, but it is too slow for the current app pipeline.

Quality judging used the original WAV audio when possible: 45 files used audio judging, 2 fell back to text-reference judging, and 3 very short clips were skipped because one provider returned empty text. Saved .txt files were treated only as historical app transcripts, not ground truth.

Raw data is retained locally in benchmarks/results/raw.json for rejudging. It is not published because it contains recording-derived transcripts.

Engine configurations

Engine	STT	Polish	Typical Latency
Standard	ElevenLabs Scribe v1	GPT-5.4 Nano	~3-5s
Fast	Groq Whisper large-v3-turbo	Groq Llama 3.1 8B	~0.5-1s
Deepgram	Deepgram Nova-3	GPT-5.4 Nano	~2-4s

Future optimizations

Optimization	Expected Impact
Deepgram Nova-3 fallback or experimental mode	Faster-than-ElevenLabs STT in some medium/long dictations
Groq LLM polishing (~200ms)	~82% polishing reduction
ElevenLabs Scribe v2 streaming	Near-zero post-stop latency
Connection pre-warming	-150ms on first call
Audio compression (Opus)	-100-700ms on slow connections

Methodology

• Python benchmark harnesses (benchmarks/bench.py, benchmarks/interfaze_groq_bench.py) using httpx async HTTP client
• WAV recordings captured via Mumbli’s debug mode
• June 2026 STT comparison used 50 real recordings, 1 STT call per provider per file, and an audio-aware judge
• Benchmark harness output is written to benchmarks/results/; recording-derived raw JSON is kept local
• All benchmarks are reproducible from source

Custom vocabulary accuracy

Tested 11 commonly mistranscribed words (proper nouns, technical terms):

Metric	Without vocabulary	With vocabulary
Exact match accuracy	36%	100%

See Custom Vocabulary for details.

Archived local experiments

Older chunking, polishing-model, and end-to-end experiments are kept in the repository under reports/ and benchmarks/results/. They are useful for implementation history, but this page reflects the current provider choice.