Contrastive Learning of Musical Representations

2021/03/17 by Janne Spijkervet, Spijkervet, Janne, John Burgoyne +1 · 10 citations
Computer Science · #Audio and Speech Processing (eess.AS) #FOS: Computer and information sciences #FOS: Electrical engineering #Machine Learning (cs.LG) #Music Technology and Sound Studies #Music and Audio Processing #Sound (cs.SD) #Speech and Audio Processing #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2103.09410

openalex publication_date 2021/03/17 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

While deep learning has enabled great advances in many areas of music, labeled music datasets remain especially hard, expensive, and time-consuming to create. In this work, we introduce SimCLR to the music domain and contribute a large chain of audio data augmentations to form a simple framework for self-supervised, contrastive learning of musical representations: CLMR. This approach works on raw time-domain music data and requires no labels to learn useful representations. We evaluate CLMR in the downstream task of music classification on the MagnaTagATune and Million Song datasets and present an ablation study to test which of our music-related innovations over SimCLR are most effective. A linear classifier trained on the proposed representations achieves a higher average precision than supervised models on the MagnaTagATune dataset, and performs comparably on the Million Song dataset. Moreover, we show that CLMR's representations are transferable using out-of-domain datasets, indicating that our method has strong generalisability in music classification. Lastly, we show that the proposed method allows data-efficient learning on smaller labeled datasets: we achieve an average precision of 33.1% despite using only 259 labeled songs in the MagnaTagATune dataset (1% of the full dataset) during linear evaluation. To foster reproducibility and future research on self-supervised learning in music, we publicly release the pre-trained models and the source code of all experiments of this paper.

Citations

• A Scale for the Measurement of the Psychological Magnitude Pitch
• Representation Learning: A Review and New Perspectives
• Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• Contrastive Multiview Coding
• Momentum Contrast for Unsupervised Visual Representation Learning
• A Simple Framework for Contrastive Learning of Visual Representations
• Big Self-Supervised Models are Strong Semi-Supervised Learners

Cited by

• Pianist Transformer: Towards Expressive Piano Performance Rendering via Scalable Self-Supervised Pre-Training
• LLM2Fx-Tools: Tool Calling For Music Post-Production
• Twenty-Five Years of MIR Research: Achievements, Practices, Evaluations, and Future Challenges
• Persian Musical Instruments Classification Using Polyphonic Data Augmentation
• Controlling Contrastive Self-Supervised Learning with Knowledge-Driven Multiple Hypothesis: Application to Beat Tracking
• VietLyrics: A Large-Scale Dataset and Models for Vietnamese Automatic Lyrics Transcription
• CVSM: Contrastive Vocal Similarity Modeling
• Representation-Based Data Quality Audits for Audio
• Codified audio language modeling learns useful representations for music information retrieval
• PESTO: Real-Time Pitch Estimation with Self-supervised Transposition-equivariant Objective

Related

• Semi-Supervised Contrastive Learning of Musical Representations
• Towards Proper Contrastive Self-supervised Learning Strategies For Music Audio Representation
• Multi-Source Contrastive Learning from Musical Audio
• MuQ: Self-Supervised Music Representation Learning with Mel Residual Vector Quantization
• CVSM: Contrastive Vocal Similarity Modeling