MUST-RAG: MUSical Text Question Answering with Retrieval Augmented Generation

2025/07/31 by Kwon, Daeyong, Doh, SeungHeon, Nam, Juhan
#Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Information Retrieval (cs.IR) #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2507.23334

Abstract

Recent advancements in Large language models (LLMs) have demonstrated remarkable capabilities across diverse domains. While they exhibit strong zero-shot performance on various tasks, LLMs' effectiveness in music-related applications remains limited due to the relatively small proportion of music-specific knowledge in their training data. To address this limitation, we propose MusT-RAG, a comprehensive framework based on Retrieval Augmented Generation (RAG) to adapt general-purpose LLMs for text-only music question answering (MQA) tasks. RAG is a technique that provides external knowledge to LLMs by retrieving relevant context information when generating answers to questions. To optimize RAG for the music domain, we (1) propose MusWikiDB, a music-specialized vector database for the retrieval stage, and (2) utilizes context information during both inference and fine-tuning processes to effectively transform general-purpose LLMs into music-specific models. Our experiment demonstrates that MusT-RAG significantly outperforms traditional fine-tuning approaches in enhancing LLMs' music domain adaptation capabilities, showing consistent improvements across both in-domain and out-of-domain MQA benchmarks. Additionally, our MusWikiDB proves substantially more effective than general Wikipedia corpora, delivering superior performance and computational efficiency.

Citations

• CLaMP 3: Universal Music Information Retrieval Across Unaligned Modalities and Unseen Languages
• The Role of Large Language Models in Musicology: Are We Ready to Trust the Machines?
• MuChoMusic: Evaluating Music Understanding in Multimodal Audio-Language Models
• The Llama 3 Herd of Models
• Enhancing Contextual Understanding in Large Language Models through Contrastive Decoding
• Large language models for biomedicine: foundations, opportunities, challenges, and best practices
• Enriching Music Descriptions with a Finetuned-LLM and Metadata for Text-to-Music Retrieval
• LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders
• ChatMusician: Understanding and Generating Music Intrinsically with LLM
• Self-Knowledge Guided Retrieval Augmentation for Large Language Models
• Music Understanding LLaMA: Advancing Text-to-Music Generation with Question Answering and Captioning
• GPT-4 Technical Report
• Toward Universal Text-to-Music Retrieval
• Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation
• Deep Bidirectional Language-Knowledge Graph Pretraining
• Contrastive Audio-Language Learning for Music
• Unsupervised Dense Information Retrieval with Contrastive Learning
• Improving language models by retrieving from trillions of tokens
• LexGLUE: A Benchmark Dataset for Legal Language Understanding in English
• LoRA: Low-Rank Adaptation of Large Language Models
• Retrieval Augmentation Reduces Hallucination in Conversation
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• Dense Passage Retrieval for Open-Domain Question Answering
• SQuAD: 100,000+ Questions for Machine Comprehension of Text
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Related

• ArtistMus: A Globally Diverse, Artist-Centric Benchmark for Retrieval-Augmented Music Question Answering
• MQAD: A Large-Scale Question Answering Dataset for Training Music Large Language Models
• Music Understanding LLaMA: Advancing Text-to-Music Generation with Question Answering and Captioning
• Exploring GPT's Ability as a Judge in Music Understanding
• Advancing the Foundation Model for Music Understanding