Controlling Contrastive Self-Supervised Learning with Knowledge-Driven Multiple Hypothesis: Application to Beat Tracking

2025/10/29 by Gagnere, Antonin, Essid, Slim, Peeters, Geoffroy
#Audio and Speech Processing (eess.AS) #FOS: Computer and information sciences #FOS: Electrical engineering #Sound (cs.SD) #electronic engineering #information engineering

paper · doi:10.48550/arxiv.2510.25560

Abstract

Ambiguities in data and problem constraints can lead to diverse, equally plausible outcomes for a machine learning task. In beat and downbeat tracking, for instance, different listeners may adopt various rhythmic interpretations, none of which would necessarily be incorrect. To address this, we propose a contrastive self-supervised pre-training approach that leverages multiple hypotheses about possible positive samples in the data. Our model is trained to learn representations compatible with different such hypotheses, which are selected with a knowledge-based scoring function to retain the most plausible ones. When fine-tuned on labeled data, our model outperforms existing methods on standard benchmarks, showcasing the advantages of integrating domain knowledge with multi-hypothesis selection in music representation learning in particular.

Citations

• S-KEY: Self-supervised Learning of Major and Minor Keys from Audio
• Beat This! Spectrograms for Beat and Downbeat Tracking
• Tempo estimation as fully self-supervised binary classification
• Music Source Separation with Band-Split RoPE Transformer
• PESTO: Pitch Estimation With Self-Supervised Transposition-Equivariant Objective
• Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothing
• Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture
• Supervised and Unsupervised Learning of Audio Representations for Music Understanding
• Audio Barlow Twins: Self-Supervised Audio Representation Learning
• Beat Transformer: Demixed Beat and Downbeat Tracking with Dilated Self-Attention
• Equivariant Self-Supervision for Musical Tempo Estimation
• Modeling Beats and Downbeats with a Time-Frequency Transformer
• Masked Siamese Networks for Label-Efficient Learning
• Masked Autoencoders Are Scalable Vision Learners
• BEiT: BERT Pre-Training of Image Transformers
• Emerging Properties in Self-Supervised Vision Transformers
• RoFormer: Enhanced Transformer with Rotary Position Embedding
• Contrastive Learning of Musical Representations
• Barlow Twins: Self-Supervised Learning via Redundancy Reduction
• Exploring Simple Siamese Representation Learning
• Contrastive Learning of General-Purpose Audio Representations
• wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
• Bootstrap your own latent: A new approach to self-supervised Learning
• Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere
• A Simple Framework for Contrastive Learning of Visual Representations
• Representation Learning with Contrastive Predictive Coding
• Learning in an Uncertain World: Representing Ambiguity Through Multiple\n Hypotheses
• Stochastic Multiple Choice Learning for Training Diverse Deep Ensembles

Related

• Self-Supervised Beat Tracking in Musical Signals with Polyphonic Contrastive Learning
• Semi-Supervised Contrastive Learning of Musical Representations
• An Experimental Comparison Of Multi-view Self-supervised Methods For Music Tagging
• Towards Proper Contrastive Self-supervised Learning Strategies For Music Audio Representation
• Multi-Source Contrastive Learning from Musical Audio