Whale: Large-Scale multilingual ASR model with w2v-BERT and E-Branchformer with large speech data

2025/06/02 by Yosuke Kashiwagi, Hayato Futami, Kashiwagi, Yosuke +5 · 2 citations
Computer Science · #Speech Recognition and Synthesis

paper · pdf · doi:10.48550/arxiv.2506.01439

Abstract

This paper reports on the development of a large-scale speech recognition model, Whale. Similar to models such as Whisper and OWSM, Whale leverages both a large model size and a diverse, extensive dataset. Whale's architecture integrates w2v-BERT self-supervised model, an encoder-decoder backbone built on E-Branchformer, and a joint CTC-attention decoding strategy. The training corpus comprises varied speech data, of not only public corpora but also in-house data, thereby enhancing the model's robustness to different speaking styles and acoustic conditions. Through evaluations on multiple benchmarks, Whale achieved comparable performance to existing models. In particular, it achieves a word error rate of 2.4% on the Librispeech test-clean set and a character error rate of 3.4% on the CSJ eval3 set, outperforming Whisper large-v3 and OWSM v3.1.

Citations

• Rapid Language Adaptation for Multilingual E2E Speech Recognition Using Encoder Prompting
• mHuBERT-147: A Compact Multilingual HuBERT Model
• OWSM-CTC: An Open Encoder-Only Speech Foundation Model for Speech Recognition, Translation, and Language Identification
• OWSM v3.1: Better and Faster Open Whisper-Style Speech Models based on E-Branchformer
• YODAS: Youtube-Oriented Dataset for Audio and Speech
• Reproducing Whisper-Style Training Using an Open-Source Toolkit and Publicly Available Data
• Scaling Speech Technology to 1,000+ Languages
• Robust Speech Recognition via Large-Scale Weak Supervision
• E-Branchformer: Branchformer with Enhanced merging for speech recognition
• FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech
• Better Intermediates Improve CTC Inference
• JTubeSpeech: corpus of Japanese speech collected from YouTube for speech recognition and speaker verification
• The People's Speech: A Large-Scale Diverse English Speech Recognition Dataset for Commercial Usage
• WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
• WenetSpeech: A 10000+ Hours Multi-domain Mandarin Corpus for Speech Recognition
• W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training
• The Multilingual TEDx Corpus for Speech Recognition and Translation
• VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation
• Common Voice: A Massively-Multilingual Speech Corpus
• Bytes are All You Need: End-to-End Multilingual Speech Recognition and Synthesis with Bytes
• ESPnet: End-to-End Speech Processing Toolkit
• Multilingual Speech Recognition With A Single End-To-End Model
• Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learning
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Cited by

• Index-MSR: A high-efficiency multimodal fusion framework for speech recognition
• Listening, Imagining & Refining: A Heuristic Optimized ASR Correction Framework with LLMs

Related

• Anatomy of Industrial Scale Multilingual ASR
• Connecting Speech Encoder and Large Language Model for ASR
• Dolphin: A Large-Scale Automatic Speech Recognition Model for Eastern Languages
• Large Language Model Guided Decoding for Self-Supervised Speech Recognition
• MSR-86K: An Evolving, Multilingual Corpus with 86,300 Hours of Transcribed Audio for Speech Recognition Research