Generalization Ability of MOS Prediction Networks

2021/10/06 by Erica Cooper, Cooper, Erica, Wen-Chin Huang +5 · 15 citations
Computer Science · #Audio and Speech Processing (eess.AS) #FOS: Electrical engineering #Speech Recognition and Synthesis #Speech and Audio Processing #Topic Modeling #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2110.02635

openalex publication_date 2021/10/06 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Automatic methods to predict listener opinions of synthesized speech remain elusive since listeners, systems being evaluated, characteristics of the speech, and even the instructions given and the rating scale all vary from test to test. While automatic predictors for metrics such as mean opinion score (MOS) can achieve high prediction accuracy on samples from the same test, they typically fail to generalize well to new listening test contexts. In this paper, using a variety of networks for MOS prediction including MOSNet and self-supervised speech models such as wav2vec2, we investigate their performance on data from different listening tests in both zero-shot and fine-tuned settings. We find that wav2vec2 models fine-tuned for MOS prediction have good generalization capability to out-of-domain data even for the most challenging case of utterance-level predictions in the zero-shot setting, and that fine-tuning to in-domain data can improve predictions. We also observe that unseen systems are especially challenging for MOS prediction models.

Cited by

• Augmenting Open-Vocabulary Dysarthric Speech Assessment with Human Perceptual Supervision
• Toward Objective and Interpretable Prosody Evaluation in Text-to-Speech: A Linguistically Motivated Approach
• Position: Towards Responsible Evaluation for Text-to-Speech
• Interpretable Audio Editing Evaluation via Chain-of-Thought Difference-Commonality Reasoning with Multimodal LLMs
• Layer-wise Analysis for Quality of Multilingual Synthesized Speech
• Improving Perceptual Audio Aesthetic Assessment via Triplet Loss and Self-Supervised Embeddings
• From Evaluation to Optimization: Neural Speech Assessment for Downstream Applications
• The AudioMOS Challenge 2025
• DRASP: A Dual-Resolution Attentive Statistics Pooling Framework for Automatic MOS Prediction
• Towards Frame-level Quality Predictions of Synthetic Speech
• Selection of Layers from Self-supervised Learning Models for Predicting Mean-Opinion-Score of Speech
• QAMRO: Quality-aware Adaptive Margin Ranking Optimization for Human-aligned Assessment of Audio Generation Systems
• Leveraging LLMs for Scalable Non-intrusive Speech Quality Assessment
• Advancing Speech Quality Assessment Through Scientific Challenges and Open-source Activities
• Feature Importance across Domains for Improving Non-Intrusive Speech Intelligibility Prediction in Hearing Aids

Related

• Improving Self-Supervised Learning-based MOS Prediction Networks
• MBNet: MOS Prediction for Synthesized Speech with Mean-Bias Network
• SAMOS: A Neural MOS Prediction Model Leveraging Semantic Representations and Acoustic Features
• LDNet: Unified Listener Dependent Modeling in MOS Prediction for Synthetic Speech
• Comparison of Speech Representations for the MOS Prediction System