> ## Documentation Index
> Fetch the complete documentation index at: https://mumbli-feat-proof-of-use.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Benchmarks — Mumbli STT Providers

> Current latency, quality, and cost benchmarks for Mumbli's speech-to-text providers: ElevenLabs, Groq, and Deepgram.

# Benchmarks

Current provider results for Mumbli's speech-to-text engines.

*Last updated: June 3, 2026*

## TL;DR

| Need                   | Pick                                           | Why                                                                                                                                        |
| ---------------------- | ---------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
| **Best quality**       | **Standard**: ElevenLabs Scribe + GPT-5.4 Nano | ElevenLabs had the most audio-judge wins: 16/50, versus 4 for Deepgram and 3 for Groq.                                                     |
| **Fastest**            | **Fast**: Groq Whisper + Groq Llama 3.1 8B     | Groq had the lowest median STT latency: 1,048ms, with the best p95 at 3,037ms.                                                             |
| **Cheapest STT**       | **Groq Whisper large-v3-turbo**                | Current listed STT price is $0.04/hour transcribed, lower than ElevenLabs Scribe at $0.22/hour and Deepgram Nova-3 at roughly \$0.29/hour. |
| **Best middle option** | **Deepgram**: Nova-3 + GPT-5.4 Nano            | Reliable 50/50 success, often faster than ElevenLabs, but weaker than ElevenLabs on judged quality.                                        |

Pricing was checked on June 3, 2026 from the official [Groq pricing](https://groq.com/pricing), [ElevenLabs API pricing](https://elevenlabs.io/pricing/api?price.section=speech_to_text), and [Deepgram pricing](https://deepgram.com/pricing) pages. Provider pricing changes; use the latency and quality results below as the durable part of the benchmark.

## Current STT benchmark

We benchmarked the latest 50 saved Mumbli recordings: 896 seconds of real dictation, ranging from 0.9s to 90.7s per clip.

| Provider           | Model                    | Success | Median STT latency | p95 latency |         Judge wins |
| ------------------ | ------------------------ | ------: | -----------------: | ----------: | -----------------: |
| Groq Whisper       | `whisper-large-v3-turbo` |   50/50 |        **1,048ms** | **3,037ms** |                  3 |
| Deepgram Nova-3    | `nova-3`                 |   50/50 |            1,928ms |     6,831ms |                  4 |
| ElevenLabs Scribe  | `scribe_v1`              |   50/50 |            2,538ms |     8,570ms |             **16** |
| Interfaze Run Task | `interfaze-beta`         |   50/50 |           22,206ms |    54,628ms |                  2 |
| Interfaze STT      | `interfaze-beta`         |   50/50 |           22,371ms |    36,445ms |                  3 |
| Tie / skipped      | —                        |       — |                  — |           — | 19 ties, 3 skipped |

**Conclusion:** ElevenLabs remains the best default transcription engine for quality. Deepgram Nova-3 is reliable and often faster than ElevenLabs, especially on longer clips, but it did not beat ElevenLabs on judged quality and sometimes returned empty text on very short clips. Groq is still the fastest option by far and remains the right engine when low latency matters more than exact wording. Interfaze worked reliably, but it is too slow for the current app pipeline.

Quality judging used the original WAV audio when possible: 45 files used audio judging, 2 fell back to text-reference judging, and 3 very short clips were skipped because one provider returned empty text. Saved `.txt` files were treated only as historical app transcripts, not ground truth.

Raw data is retained locally in `benchmarks/results/raw.json` for rejudging. It is not published because it contains recording-derived transcripts.

## Engine configurations

| Engine       | STT                         | Polish            | Typical Latency |
| ------------ | --------------------------- | ----------------- | --------------- |
| **Standard** | ElevenLabs Scribe v1        | GPT-5.4 Nano      | \~3-5s          |
| **Fast**     | Groq Whisper large-v3-turbo | Groq Llama 3.1 8B | \~0.5-1s        |
| **Deepgram** | Deepgram Nova-3             | GPT-5.4 Nano      | \~2-4s          |

## Future optimizations

| Optimization                                  | Expected Impact                                           |
| --------------------------------------------- | --------------------------------------------------------- |
| Deepgram Nova-3 fallback or experimental mode | Faster-than-ElevenLabs STT in some medium/long dictations |
| Groq LLM polishing (\~200ms)                  | \~82% polishing reduction                                 |
| ElevenLabs Scribe v2 streaming                | Near-zero post-stop latency                               |
| Connection pre-warming                        | -150ms on first call                                      |
| Audio compression (Opus)                      | -100-700ms on slow connections                            |

## Methodology

* Python benchmark harnesses (`benchmarks/bench.py`, `benchmarks/interfaze_groq_bench.py`) using `httpx` async HTTP client
* WAV recordings captured via Mumbli's debug mode
* June 2026 STT comparison used 50 real recordings, 1 STT call per provider per file, and an audio-aware judge
* Benchmark harness output is written to `benchmarks/results/`; recording-derived raw JSON is kept local
* All benchmarks are [reproducible from source](https://github.com/fireharp/mumbli/tree/main/benchmarks)

## Custom vocabulary accuracy

Tested 11 commonly mistranscribed words (proper nouns, technical terms):

| Metric               | Without vocabulary | With vocabulary |
| -------------------- | ------------------ | --------------- |
| Exact match accuracy | 36%                | **100%**        |

See [Custom Vocabulary](/custom-vocabulary-for-dictation) for details.

## Archived local experiments

Older chunking, polishing-model, and end-to-end experiments are kept in the repository under `reports/` and `benchmarks/results/`. They are useful for implementation history, but this page reflects the current provider choice.