The VoiceMOS Challenge 2022

2022/03/21 by Wen-Chin Huang, Huang, Wen-Chin, Erica Cooper +9 · 27 citations
Computer Science · Engineering · #Audio and Speech Processing (eess.AS) #FOS: Computer and information sciences #FOS: Electrical engineering #Sound (cs.SD) #cs.SD #eess.AS #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2203.11389

Accepted to Interspeech 2022

arxiv created 2022/07/04 · arxiv updated 2022/07/05

Abstract

We present the first edition of the VoiceMOS Challenge, a scientific event that aims to promote the study of automatic prediction of the mean opinion score (MOS) of synthetic speech. This challenge drew 22 participating teams from academia and industry who tried a variety of approaches to tackle the problem of predicting human ratings of synthesized speech. The listening test data for the main track of the challenge consisted of samples from 187 different text-to-speech and voice conversion systems spanning over a decade of research, and the out-of-domain track consisted of data from more recent systems rated in a separate listening test. Results of the challenge show the effectiveness of fine-tuning self-supervised speech models for the MOS prediction task, as well as the difficulty of predicting MOS ratings for unseen speakers and listeners, and for unseen systems in the out-of-domain setting.

Cited by

• The T12 System for AudioMOS Challenge 2025: Audio Aesthetics Score Prediction System Using KAN- and VERSA-based Models
• Augmenting Open-Vocabulary Dysarthric Speech Assessment with Human Perceptual Supervision
• NaturalVoices: A Large-Scale, Spontaneous and Emotional Podcast Dataset for Voice Conversion
• From Scores to Preferences: Redefining MOS Benchmarking for Speech Quality Reward Modeling
• Assessing speech quality metrics for evaluation of neural audio codecs under clean speech conditions
• Objective Evaluation of Prosody and Intelligibility in Speech Synthesis via Conditional Prediction of Discrete Tokens
• LatentVoiceGrad: Nonparallel Voice Conversion with Latent Diffusion/Flow-Matching Models
• Improving Perceptual Audio Aesthetic Assessment via Triplet Loss and Self-Supervised Embeddings
• From Evaluation to Optimization: Neural Speech Assessment for Downstream Applications
• The AudioMOS Challenge 2025
• Towards Frame-level Quality Predictions of Synthetic Speech
• Selection of Layers from Self-supervised Learning Models for Predicting Mean-Opinion-Score of Speech
• Leveraging LLMs for Scalable Non-intrusive Speech Quality Assessment
• Fine-Tuning Text-to-Speech Diffusion Models Using Reinforcement Learning with Human Feedback
• Advancing Speech Quality Assessment Through Scientific Challenges and Open-source Activities
• Unifying Listener Scoring Scales: Comparison Learning Framework for Speech Quality Assessment and Continuous Speech Emotion Recognition
• Differentiable Reward Optimization for LLM based TTS system
• URGENT-PK: Perceptually-Aligned Ranking Model Designed for Speech Enhancement Competition
• Post-training for Deepfake Speech Detection
• Rethinking Mean Opinion Scores in Speech Quality Assessment: Aggregation through Quantized Distribution Fitting
• JIS: A Speech Corpus of Japanese Idol Speakers with Various Speaking Styles
• TTSOps: A Closed-Loop Corpus Optimization Framework for Training Multi-Speaker TTS Models from Dark Data
• Improving Speech Enhancement with Multi-Metric Supervision from Learned Quality Assessment
• SALF-MOS: Speaker Agnostic Latent Features Downsampled for MOS Prediction
• ARECHO: Autoregressive Evaluation via Chain-Based Hypothesis Optimization for Speech Multi-Metric Estimation
• Uni-VERSA: Versatile Speech Assessment with a Unified Network
• SHEET: A Multi-purpose Open-source Speech Human Evaluation Estimation Toolkit

Related

• The VoiceMOS Challenge 2023: Zero-shot Subjective Speech Quality Prediction for Multiple Domains
• The VoiceMOS Challenge 2024: Beyond Speech Quality Prediction
• UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022
• The T05 System for The VoiceMOS Challenge 2024: Transfer Learning from Deep Image Classifier to Naturalness MOS Prediction of High-Quality Synthetic Speech
• The ZevoMOS entry to VoiceMOS Challenge 2022