Assessing speech quality metrics for evaluation of neural audio codecs under clean speech conditions

2025/09/29 by Mack, Wolfgang, Topaloglu, Nezih, Lechler, Laura +4 · 1 citation
#Audio and Speech Processing (eess.AS) #FOS: Electrical engineering #electronic engineering #information engineering

paper · doi:10.48550/arxiv.2509.24457

Abstract

Objective speech-quality metrics are widely used to assess codec performance. However, for neural codecs, it is often unclear which metrics provide reliable quality estimates. To address this, we evaluated 45 objective metrics by correlating their scores with subjective listening scores for clean speech across 17 codec conditions. Neural-based metrics such as scoreq and utmos achieved the highest Pearson correlations with subjective scores. Further analysis across different subjective quality ranges revealed that non-intrusive metrics tend to saturate at high subjective quality levels.

Citations

• MUSHRA-1S: A scalable and sensitive test approach for evaluating top-tier speech processing systems
• Crowdsourcing MUSHRA Tests in the Age of Generative Speech Technologies: A Comparative Analysis of Subjective and Objective Testing Methods
• DualCodec: A Low-Frame-Rate, Semantically-Enhanced Neural Audio Codec for Speech Generation
• VERSA: A Versatile Evaluation Toolkit for Speech, Audio, and Music
• Scaling Transformers for Low-Bitrate High-Quality Speech Coding
• MOS-Bench: Benchmarking Generalization Abilities of Subjective Speech Quality Assessment Models
• SNAC: Multi-Scale Neural Audio Codec
• SCOREQ: Speech Quality Assessment with Contrastive Regression
• Moshi: a speech-text foundation model for real-time dialogue
• The T05 System for The VoiceMOS Challenge 2024: Transfer Learning from Deep Image Classifier to Naturalness MOS Prediction of High-Quality Synthetic Speech
• The VoiceMOS Challenge 2024: Beyond Speech Quality Prediction
• BigCodec: Pushing the Limits of Low-Bitrate Neural Speech Codec
• The PESQetarian: On the Relevance of Goodhart's Law for Speech Enhancement
• ESC: Efficient Speech Coding with Cross-Scale Residual Vector Quantized Transformers
• SpeechBERTScore: Reference-Aware Automatic Evaluation of Speech Generation Leveraging NLP Evaluation Metrics
• ESPnet-SPK: full pipeline speaker embedding toolkit with reproducible recipes, self-supervised front-ends, and off-the-shelf models
• The VoiceMOS Challenge 2023: Zero-shot Subjective Speech Quality Prediction for Multiple Domains
• Multi-dimensional Speech Quality Assessment in Crowdsourcing
• High-Fidelity Audio Compression with Improved RVQGAN
• PLCMOS -- a data-driven non-intrusive metric for the evaluation of packet loss concealment algorithms
• TorchAudio-Squim: Reference-less Speech Quality and Intelligibility measures in TorchAudio
• High Fidelity Neural Audio Compression
• ESPnet-SE++: Speech Enhancement for Robust Speech Recognition, Translation, and Understanding
• UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022
• The VoiceMOS Challenge 2022
• NORESQA: A Framework for Speech Quality Assessment using Non-Matching References
• WARP-Q: Quality Prediction For Generative Neural Speech Codecs
• ESPnet-TTS: Unified, Reproducible, and Integratable Open Source End-to-End Text-to-Speech Toolkit

Cited by

• Low-Resource Audio Codec (LRAC): 2025 Challenge Description
• MUSHRA-1S: A scalable and sensitive test approach for evaluating top-tier speech processing systems

Related

• Evaluating Objective Speech Quality Metrics for Neural Audio Codecs
• On the Relation Between Speech Quality and Quantized Latent Representations of Neural Codecs
• WARP-Q: Quality Prediction For Generative Neural Speech Codecs
• Non-intrusive speech quality assessment using neural networks
• DNSMOS P.835: A Non-Intrusive Perceptual Objective Speech Quality Metric to Evaluate Noise Suppressors