RELATE: Subjective evaluation dataset for automatic evaluation of relevance between text and audio

2025/06/30 by Kanamori, Yusuke, Okamoto, Yuki, Takano, Taisei +3 · 2 citations
#Audio and Speech Processing (eess.AS) #FOS: Computer and information sciences #FOS: Electrical engineering #Sound (cs.SD) #electronic engineering #information engineering

paper · doi:10.48550/arxiv.2506.23582

Abstract

In text-to-audio (TTA) research, the relevance between input text and output audio is an important evaluation aspect. Traditionally, it has been evaluated from both subjective and objective perspectives. However, subjective evaluation is costly in terms of money and time, and objective evaluation is unclear regarding the correlation to subjective evaluation scores. In this study, we construct RELATE, an open-sourced dataset that subjectively evaluates the relevance. Also, we benchmark a model for automatically predicting the subjective evaluation score from synthesized audio. Our model outperforms a conventional CLAPScore model, and that trend extends to many sound categories.

Citations

• Challenge on Sound Scene Synthesis: Evaluating Text-to-Audio Generation
• The VoiceMOS Challenge 2024: Beyond Speech Quality Prediction
• Tango 2: Aligning Diffusion-based Text-to-Audio Generations through Direct Preference Optimization
• Subjective-Aligned Dataset and Metric for Text-to-Video Quality Assessment
• PAM: Prompting Audio-Language Models for Audio Quality Assessment
• AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining
• Environmental sound synthesis from vocal imitations and sound event labels
• Toward Verifiable and Reproducible Human Evaluation for Text-to-Image Generation
• Audio-Text Models Do Not Yet Leverage Natural Language
• Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models
• AudioLDM: Text-to-Audio Generation with Latent Diffusion Models
• Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation
• Diffsound: Discrete Diffusion Model for Text-to-sound Generation
• UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022
• WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
• LDNet: Unified Listener Dependent Modeling in MOS Prediction for Synthetic Speech
• MBNet: MOS Prediction for Synthesized Speech with Mean-Bias Network
• wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
• Class-Balanced Loss Based on Effective Number of Samples
• Adam: A Method for Stochastic Optimization
• Use of Ranks in One-Criterion Variance Analysis
• On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other

Cited by

• AudioBERTScore: Objective Evaluation of Environmental Sound Synthesis Based on Similarity of Audio embedding Sequences
• Human-CLAP: Human-perception-based contrastive language-audio pretraining

Related

• AudioEval: Automatic Dual-Perspective and Multi-Dimensional Evaluation of Text-to-Audio-Generation
• AudioBERTScore: Objective Evaluation of Environmental Sound Synthesis Based on Similarity of Audio embedding Sequences
• TTA-Bench: A Comprehensive Benchmark for Evaluating Text-to-Audio Models
• Crowdsourcing and Evaluating Text-Based Audio Retrieval Relevances
• BATON: Aligning Text-to-Audio Model with Human Preference Feedback