Contrastive timbre representations for musical instrument and synthesizer retrieval

2025/09/16 by Gwendal Le Vaillant, Vaillant, Gwendal Le, Yannick Molle +1
Arts and Humanities · Computer Science · #Artificial Intelligence (cs.AI) #Diverse Musicological Studies #FOS: Computer and information sciences #Music Technology and Sound Studies #Music and Audio Processing #Sound (cs.SD)

paper · pdf · doi:10.48550/arxiv.2509.13285

openalex publication_date 2025/09/16 · openalex created_date 2025/10/18 · openalex updated_date 2026/07/28

Abstract

Efficiently retrieving specific instrument timbres from audio mixtures remains a challenge in digital music production. This paper introduces a contrastive learning framework for musical instrument retrieval, enabling direct querying of instrument databases using a single model for both single- and multi-instrument sounds. We propose techniques to generate realistic positive/negative pairs of sounds for virtual musical instruments, such as samplers and synthesizers, addressing limitations in common audio data augmentation methods. The first experiment focuses on instrument retrieval from a dataset of 3,884 instruments, using single-instrument audio as input. Contrastive approaches are competitive with previous works based on classification pre-training. The second experiment considers multi-instrument retrieval with a mixture of instruments as audio input. In this case, the proposed contrastive framework outperforms related works, achieving 81.7% top-1 and 95.7% top-5 accuracies for three-instrument mixtures.

Citations

• Asymmetric Clean Segments-Guided Self-Supervised Learning for Robust Speaker Verification
• Listen, Think, and Understand
• Multi-Source Contrastive Learning from Musical Audio
• Show Me the Instruments: Musical Instrument Retrieval from Mixture Audio
• Hybrid Transformers for Music Source Separation
• Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation
• Hybrid Spectrogram and Waveform Source Separation
• AST: Audio Spectrogram Transformer
• Contrastive Learning of General-Purpose Audio Representations
• A Simple Framework for Contrastive Learning of Visual Representations
• Cutting Music Source Separation Some Slakh: A Dataset to Study the Impact of Training Data Quality and Quantity
• Representation Learning with Contrastive Predictive Coding
• Unsupervised Learning of Semantic Audio Representations
• Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders

Related

• Show Me the Instruments: Musical Instrument Retrieval from Mixture Audio
• Fx-Encoder++: Extracting Instrument-Wise Audio Effects Representations from Mixtures
• Multi-Source Contrastive Learning from Musical Audio
• Exploring Isolated Musical Notes as Pre-training Data for Predominant Instrument Recognition in Polyphonic Music
• Contrastive Learning from Synthetic Audio Doppelgängers