GD-Retriever: Controllable Generative Text-Music Retrieval with Diffusion Models

2025/06/22 by Guinot, Julien, Quinton, Elio, Fazekas, György · 1 citation
#Audio and Speech Processing (eess.AS) #FOS: Computer and information sciences #FOS: Electrical engineering #Sound (cs.SD) #electronic engineering #information engineering

paper · doi:10.48550/arxiv.2506.17886

Abstract

Multimodal contrastive models have achieved strong performance in text-audio retrieval and zero-shot settings, but improving joint embedding spaces remains an active research area. Less attention has been given to making these systems controllable and interactive for users. In text-music retrieval, the ambiguity of freeform language creates a many-to-many mapping, often resulting in inflexible or unsatisfying results. We introduce Generative Diffusion Retriever (GDR), a novel framework that leverages diffusion models to generate queries in a retrieval-optimized latent space. This enables controllability through generative tools such as negative prompting and denoising diffusion implicit models (DDIM) inversion, opening a new direction in retrieval control. GDR improves retrieval performance over contrastive teacher models and supports retrieval in audio-only latent spaces using non-jointly trained encoders. Finally, we demonstrate that GDR enables effective post-hoc manipulation of retrieval behavior, enhancing interactive control for text-music retrieval tasks.

Citations

• DiffGAP: A Lightweight Diffusion Module in Contrastive Space for Bridging Cross-Model Gap
• Leave-One-EquiVariant: Alleviating invariance-related information loss in contrastive music representations
• DRCap: Decoding CLAP Latents with Retrieval-Augmented Generation for Zero-shot Audio Captioning
• CoLLAP: Contrastive Long-form Language-Audio Pretraining with Musical Temporal Structure Augmentation
• Prompt Sliders for Fine-Grained Control, Editing and Erasing of Concepts in Diffusion Models
• Augment, Drop & Swap: Improving Diversity in LLM Captions for Efficient Music-Text Representation Learning
• SpecMaskGIT: Masked Generative Modeling of Audio Spectrograms for Efficient Audio Synthesis and Beyond
• T-CLAP: Temporal-Enhanced Contrastive Language-Audio Pretraining
• Long-form music generation with latent diffusion
• Continuous, Subject-Specific Attribute Control in T2I Models by Identifying Semantic Directions
• Arrange, Inpaint, and Refine: Steerable Long-term Music Audio Generation and Editing via Content-based Controls
• Cacophony: An Improved Contrastive Audio-Text Model
• MusicMagus: Zero-Shot Text-to-Music Editing via Diffusion Models
• Fast Timing-Conditioned Latent Audio Diffusion
• DITTO: Diffusion Inference-Time T-Optimization for Music Generation
• Improving fine-grained understanding in image-text pre-training
• Similar but Faster: Manipulation of Tempo in Music Audio Embeddings for Tempo Prediction and Search
• Compositional Inversion for Stable Diffusion Models
• The Song Describer Dataset: a Corpus of Audio Captions for Music-and-Language Evaluation
• Mustango: Toward Controllable Text-to-Music Generation
• Music ControlNet: Multiple Time-varying Controls for Music Generation
• Dynamic Prompt Learning: Addressing Cross-Attention Leakage for Text-Based Image Editing
• RECAP: Retrieval-Augmented Audio Captioning
• Selecting which Dense Retriever to use for Zero-Shot Search
• AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining
• MusicLDM: Enhancing Novelty in Text-to-Music Generation Using Beat-Synchronous Mixup Strategies
• Simple and Controllable Music Generation
• Negative-prompt Inversion: Fast Image Inversion for Editing with Text-guided Diffusion Models
• Prompt Tuning Inversion for Text-Driven Image Editing Using Diffusion Models
• Sigmoid Loss for Language Image Pre-Training
• Test-Time Distribution Normalization for Contrastively Learned Vision-language Models
• AudioLDM: Text-to-Audio Generation with Latent Diffusion Models
• MuChoMusic dataset
• Scalable Diffusion Models with Transformers
• Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation
• Scaling Instruction-Finetuned Language Models
• Supervised and Unsupervised Learning of Audio Representations for Music Understanding
• The Vendi Score: A Diversity Evaluation Metric for Machine Learning
• MuLan: A Joint Embedding of Music Audio and Natural Language
• Contrastive Audio-Language Learning for Music
• An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion
• Prompt-to-Prompt Image Editing with Cross Attention Control
• Classifier-Free Diffusion Guidance
• Scaling Autoregressive Models for Content-Rich Text-to-Image Generation
• Hierarchical Text-Conditional Image Generation with CLIP Latents
• HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detection
• High-Resolution Image Synthesis with Latent Diffusion Models
• High-Resolution Image Synthesis with Latent Diffusion Models
• Learning Transferable Visual Models From Natural Language Supervision
• Taming Transformers for High-Resolution Image Synthesis
• Denoising Diffusion Probabilistic Models
• Disentangled Multidimensional Metric Learning for Music Similarity
• A Simple Framework for Contrastive Learning of Visual Representations
• RoBERTa: A Robustly Optimized BERT Pretraining Approach
• Correlation Alignment for Unsupervised Domain Adaptation
• Moûsai: Text-to-Music Generation with Long-Context Latent Diffusion

Cited by

• Controllable Embedding Transformation for Mood-Guided Music Retrieval

Related

• DiffATR: Diffusion-based Generative Modeling for Audio-Text Retrieval
• DiffuGR: Generative Document Retrieval with Diffusion Language Models
• Mustango: Toward Controllable Text-to-Music Generation
• ERNIE-Music: Text-to-Waveform Music Generation with Diffusion Models
• DiffusionRet: Generative Text-Video Retrieval with Diffusion Model