Open ASR Leaderboard: Towards Reproducible and Transparent Multilingual and Long-Form Speech Recognition Evaluation

2025/10/08 by Vaibhav Srivastav, Srivastav, Vaibhav, Zheng, Steven +11
Computer Science · #Artificial Intelligence (cs.AI) #Audio and Speech Processing (eess.AS) #Computation and Language (cs.CL) #FOS: Computer and information sciences #FOS: Electrical engineering #Sound (cs.SD) #Speech Recognition and Synthesis #Speech and Audio Processing #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2510.06961

openalex publication_date 2025/10/08 · openalex created_date 2025/10/13 · openalex updated_date 2026/07/31

Abstract

We present the Open ASR Leaderboard, a reproducible benchmarking platform with community contributions from academia and industry. It compares 86 open-source and proprietary systems across 12 datasets, with English short- and long-form and multilingual short-form tracks. We standardize word error rate (WER) and inverse real-time factor (RTFx) evaluation for consistent accuracy-efficiency comparisons across model architectures and toolkits (e.g., ESPNet, NeMo, SpeechBrain, Transformers). We observe that Conformer-based encoders paired with transformer-based decoders achieve the best average WER, while connectionist temporal classification (CTC) and token-and-duration transducer (TDT) decoders offer superior RTFx, making them better suited for long-form and batched processing. All code and dataset loaders are open-sourced to support transparent, extensible evaluation. We present our evaluation methodology to facilitate community-driven benchmarking in ASR and other tasks.

Citations

• Qwen3-ASR Technical Report
• Omnilingual ASR: Open-Source Multilingual Speech Recognition for 1600+ Languages
• Streaming Sequence-to-Sequence Learning with Delayed Streams Modeling
• OWSM v4: Improving Open Whisper-Style Speech Models via Data Scaling and Cleaning
• Evaluation of LLMs in Speech is Often Flawed: Test Set Contamination in Large Language Models for Speech Recognition
• LiteASR: Efficient Automatic Speech Recognition with Low-Rank Approximation
• Moonshine: Speech Recognition for Live Transcription and Voice Commands
• CrisperWhisper: Accurate Timestamps on Verbatim Speech Transcriptions
• Open-Source Conversational AI with SpeechBrain 1.0
• Distil-Whisper: Robust Knowledge Distillation via Large-Scale Pseudo Labelling
• Investigating End-to-End ASR Architectures for Long Form Audio Transcription
• Fast Conformer with Linearly Scalable Attention for Efficient Speech Recognition
• Efficient Sequence Transduction by Jointly Predicting Tokens and Durations
• Robust Speech Recognition via Large-Scale Weak Supervision
• FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech
• Earnings-22: A Practical Benchmark for Accents in the Wild
• data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language
• Datasets: A Community Library for Natural Language Processing
• HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units
• SPGISpeech: 5,000 hours of transcribed financial audio for fully\n formatted end-to-end speech recognition
• VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation
• CoVoST 2 and Massively Multilingual Speech-to-Text Translation
• wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
• Conformer: Convolution-augmented Transformer for Speech Recognition
• Conformer: Convolution-augmented Transformer for Speech Recognition
• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• NeMo: a toolkit for building AI applications using Neural Modules
• ESPnet: End-to-End Speech Processing Toolkit

Cited by

• Whisfusion: Parallel ASR Decoding with Masked Diffusion

Related

• MediaSpeech: Multilanguage ASR Benchmark and Dataset
• Anatomy of Industrial Scale Multilingual ASR
• The ML-SUPERB 2.0 Challenge: Towards Inclusive ASR Benchmarking for All Language Varieties
• Advocating Character Error Rate for Multilingual ASR Evaluation
• Less is More: Accurate Speech Recognition & Translation without Web-Scale Data