Benchmarks
Benchmarks for small languages, with sources.
Benchmarks for speech recognition and machine translation in small European languages, collected by DeepTranslate. Each result says whether it comes from our lab, from production or from a public study, whether it measures speech or translation, and where the number comes from.
Last updated 19 October 2026
How to read the results
Lab
Tested by us
Production
Live for a client
Speech
WER
Translation
Quality score
Speech and translation results for small languages
| Language | Task | System | Result | Type | Source |
|---|---|---|---|---|---|
| Flemish | Task: Speech, WER | System: Deepgram | Result: 9.1% | Type: Lab | Source: DeepTranslate research, Oct 2026 |
| Flemish | Task: Speech, WER | System: NeLF | Result: 15.4% | Type: Lab | Source: DeepTranslate research, Oct 2026 |
| Norwegian Nynorsk | Task: Speech, WER | System: NB-Whisper Large Verbatim | Result: 17.8% conversation, 14.1% read | Type: Lab | Source: DeepTranslate research, Oct 2026 |
| Norwegian Nynorsk | Task: Speech, WER | System: NB-Whisper Large | Result: 37.1% conversation, 12.3% read | Type: Lab | Source: DeepTranslate research, Oct 2026 |
| Swiss German | Task: Speech, WER | System: Whisper large-v3, zero-shot | Result: 23 to 29.5% | Type: Public | Source: Does Whisper Understand Swiss German?, 2024 |
| Luxembourgish | Task: Speech, WER | System: Lux-ASR | Result: 10 to 13% | Type: Public | Source: University of Luxembourg, Lux-ASR project |
| Basque | Task: Speech, WER | System: Whisper large-v2 | Result: 43% | Type: Public | Source: arXiv 2503.23542, Common Voice |
| Spanish, for reference | Task: Speech, WER | System: Whisper large-v2 | Result: 5% | Type: Public | Source: arXiv 2503.23542, Common Voice |
| Luxembourgish to FR, DE, EN | Task: Translation, editor score | System: DeepTranslate engine | Result: No reviewed segment below 3 of 5 | Type: Production | Source: RTL Lëtzebuerg, 2026 |
Results from different test sets are not directly comparable. Each line links to its source and method.
Why measure small languages separately?
1 in 10
machine translations for low-resource pairs detach from the source entirely (Guerreiro et al., TACL 2023)
43% vs 5%
Whisper word error rate on Basque vs Spanish, same model, same benchmark
Frequently asked questions
What is word error rate (WER)?
WER counts the words a speech system changes, drops or adds compared with a reference transcript. Lower is better. It is not a share of false facts and does not measure editing time.
Can results from different benchmarks be compared?
Only with care. Test sets differ in speakers, audio and style. Read speech scores much better than conversation: in our Nynorsk test the same model scored 12.3% on read speech and 37.1% on conversation.