AudioBERTScore: Objective Evaluation of Environmental Sound Synthesis Based on Similarity of Audio embedding Sequences

2025/07/01 by M Kishi, Kishi, Minoru, Ryosuke Sakai +7
Computer Science · #Audio and Speech Processing (eess.AS) #FOS: Computer and information sciences #FOS: Electrical engineering #Music Technology and Sound Studies #Music and Audio Processing #Sound (cs.SD) #Speech and Audio Processing #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2507.00475

openalex publication_date 2025/07/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We propose a novel objective evaluation metric for synthesized audio in text-to-audio (TTA), aiming to improve the performance of TTA models. In TTA, subjective evaluation of the synthesized sound is an important, but its implementation requires monetary costs. Therefore, objective evaluation such as mel-cepstral distortion are used, but the correlation between these objective metrics and subjective evaluation values is weak. Our proposed objective evaluation metric, AudioBERTScore, calculates the similarity between embedding of the synthesized and reference sounds. The method is based not only on the max-norm used in conventional BERTScore but also on the p-norm to reflect the non-local nature of environmental sounds. Experimental results show that scores obtained by the proposed method have a higher correlation with subjective evaluation values than conventional metrics.

Citations

• Human-CLAP: Human-perception-based contrastive language-audio pretraining
• RELATE: Subjective evaluation dataset for automatic evaluation of relevance between text and audio
• A Reference-free Metric for Language-Queried Audio Source Separation using Contrastive Language-Audio Pretraining
• Leveraging AI to Generate Audio for User-generated Content in Video Games
• PAM: Prompting Audio-Language Models for Audio Quality Assessment
• SpeechBERTScore: Reference-Aware Automatic Evaluation of Speech Generation Leveraging NLP Evaluation Metrics
• AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining
• Self-supervised Audio Teacher-Student Transformer for Both Clip-level and Frame-level Tasks
• Foley Sound Synthesis at the DCASE 2023 Challenge
• AudioLDM: Text-to-Audio Generation with Latent Diffusion Models
• AudioGen: Textually Guided Audio Generation
• How Should We Evaluate Synthesized Environmental Sounds
• HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units
• AST: Audio Spectrogram Transformer
• WARP-Q: Quality Prediction For Generative Neural Speech Codecs
• BERTScore: Evaluating Text Generation with BERT
• Fréchet Audio Distance: A Metric for Evaluating Music Enhancement Algorithms
• Attention Is All You Need
• Backpropagation Applied to Handwritten Zip Code Recognition
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Related

• Challenge on Sound Scene Synthesis: Evaluating Text-to-Audio Generation
• How Should We Evaluate Synthesized Environmental Sounds
• RELATE: Subjective evaluation dataset for automatic evaluation of relevance between text and audio
• PAM: Prompting Audio-Language Models for Audio Quality Assessment
• Evaluating generative audio systems and their metrics