OWSM v4: Improving Open Whisper-Style Speech Models via Data Scaling and Cleaning

2025/05/31 by Yifan Peng, Shakeel Muhammad, Peng, Yifan +11 · 13 citations
Computer Science · #Audio and Speech Processing (eess.AS) #Computation and Language (cs.CL) #FOS: Computer and information sciences #FOS: Electrical engineering #Sound (cs.SD) #Speech Recognition and Synthesis #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2506.00338

openalex publication_date 2025/05/31 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The Open Whisper-style Speech Models (OWSM) project has developed a series of fully open speech foundation models using academic-scale resources, but their training data remains insufficient. This work enhances OWSM by integrating YODAS, a large-scale web-crawled dataset with a Creative Commons license. However, incorporating YODAS is nontrivial due to its wild nature, which introduces challenges such as incorrect language labels and audio-text misalignments. To address this, we develop a scalable data-cleaning pipeline using public toolkits, yielding a dataset with 166,000 hours of speech across 75 languages. Our new series of OWSM v4 models, trained on this curated dataset alongside existing OWSM data, significantly outperform previous versions on multilingual benchmarks. Our models even match or surpass frontier industrial models like Whisper and MMS in multiple scenarios. We will publicly release the cleaned YODAS data, pre-trained models, and all associated scripts via the ESPnet toolkit.

Cited by

• MoLGE: Mixture of Language Group Experts for Efficient Scaling of Massively Multilingual Speech Recognition
• Hearing to Translate: The Effectiveness of Speech Modality Integration into LLMs
• Group-Aware Partial Model Merging for Children's Automatic Speech Recognition
• POWSM: A Phonetic Open Whisper-Style Speech Foundation Model
• Adapting Speech Foundation Models with Large Language Models for Unified Speech Recognition
• Data-Centric Lessons To Improve Speech-Language Pretraining
• Chain-of-Thought Reasoning in Streaming Full-Duplex End-to-End Spoken Dialogue Systems
• Stream RAG: Instant and Accurate Spoken Dialogue Systems with Streaming Tool Usage
• Open ASR Leaderboard: Towards Reproducible and Transparent Multilingual and Long-Form Speech Recognition Evaluation
• Evaluation of preprocessing pipelines in the creation of in-the-wild TTS datasets
• Geolocation-Aware Robust Spoken Language Identification
• CAMÕES: A Comprehensive Automatic Speech Recognition Benchmark for European Portuguese
• MiDashengLM: Efficient Audio Understanding with General Audio Captions

Related

• Reproducing Whisper-Style Training Using an Open-Source Toolkit and Publicly Available Data
• OWSM v3.1: Better and Faster Open Whisper-Style Speech Models based on E-Branchformer
• On the Effects of Heterogeneous Data Sources on Speech-to-Text Foundation Models
• FAMA: The First Large-Scale Open-Science Speech Foundation Model for English and Italian
• OpusLM: A Family of Open Unified Speech Language Models