Music Understanding LLaMA: Advancing Text-to-Music Generation with Question Answering and Captioning

2023/08/22 by Shansong Liu, Atin Sakkeer Hussain, Liu, Shansong +5 · 20 citations
Computer Science · #Artificial Intelligence (cs.AI) #Audio and Speech Processing (eess.AS) #Computation and Language (cs.CL) #FOS: Computer and information sciences #FOS: Electrical engineering #Multimedia (cs.MM) #Music Technology and Sound Studies #Music and Audio Processing #Sound (cs.SD) #Speech Recognition and Synthesis #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2308.11276

openalex publication_date 2023/08/22 · openalex created_date 2023/08/24 · openalex updated_date 2026/07/28

Abstract

Text-to-music generation (T2M-Gen) faces a major obstacle due to the scarcity of large-scale publicly available music datasets with natural language captions. To address this, we propose the Music Understanding LLaMA (MU-LLaMA), capable of answering music-related questions and generating captions for music files. Our model utilizes audio representations from a pretrained MERT model to extract music features. However, obtaining a suitable dataset for training the MU-LLaMA model remains challenging, as existing publicly accessible audio question answering datasets lack the necessary depth for open-ended music question answering. To fill this gap, we present a methodology for generating question-answer pairs from existing audio captioning datasets and introduce the MusicQA Dataset designed for answering open-ended music-related questions. The experiments demonstrate that the proposed MU-LLaMA model, trained on our designed MusicQA dataset, achieves outstanding performance in both music question answering and music caption generation across various metrics, outperforming current state-of-the-art (SOTA) models in both fields and offering a promising advancement in the T2M-Gen research field.

Cited by

• Revisiting Audio-language Pretraining for Learning General-purpose Audio Representation
• Learning to Hear by Seeing: It's Time for Vision Language Models to Understand Artistic Emotion from Sight and Sound
• Towards General Auditory Intelligence: Large Multimodal Models for Machine Listening and Speaking
• Assessing Factual Music Comprehension in Large Audio Language Models
• InteractiveOmni: A Unified Omni-modal Model for Audio-Visual Multi-turn Dialogue
• AURA Score: A Metric For Holistic Audio Question Answering Evaluation
• ABC-Eval: Benchmarking Large Language Models on Symbolic Music Understanding and Instruction Following
• Investigating Modality Contribution in Audio LLMs for Music
• A Survey on Evaluation Metrics for Music Generation
• Jamendo-QA: A Large-Scale Music Question Answering Dataset
• Competitive Audio-Language Models with Data-Efficient Single-Stage Training on Public Data
• MQAD: A Large-Scale Question Answering Dataset for Training Music Large Language Models
• Cryfish: On deep audio analysis with Large Language Models
• Audio Flamingo Sound-CoT Technical Report: Improving Chain-of-Thought Reasoning in Sound Understanding
• MiDashengLM: Efficient Audio Understanding with General Audio Captions
• SpeechR: A Benchmark for Speech Reasoning in Large Audio-Language Models
• MeshLLM: Empowering Large Language Models to Progressively Understand and Generate 3D Mesh
• Advancing the Foundation Model for Music Understanding
• MUST-RAG: MUSical Text Question Answering with Retrieval Augmented Generation
• DIFFA: Large Language Diffusion Models Can Listen and Understand

Related

• MQAD: A Large-Scale Question Answering Dataset for Training Music Large Language Models
• MuMu-LLaMA: Multi-modal Music Understanding and Generation via Large Language Models
• LP-MusicCaps: LLM-Based Pseudo Music Captioning
• Music's Multimodal Complexity in AVQA: Why We Need More than General Multimodal LLMs
• MUST-RAG: MUSical Text Question Answering with Retrieval Augmented Generation