MATPAC++: Enhanced Masked Latent Prediction for Self-Supervised Audio Representation Learning

2025/08/18 by Aurian Quélennec, Aurian Quelennec, Quelennec, Aurian +6 · 1 voice
Computer Science · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Music and Audio Processing #Natural Language Processing Techniques #Sound (cs.SD) #Speech Recognition and Synthesis #cs.AI #cs.SD

paper · pdf · doi:10.48550/arxiv.2508.12709

openalex publication_date 2025/08/18 · arxiv published 2025/08/18 · arxiv updated 2025/08/18 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Masked latent prediction has emerged as a leading paradigm in self-supervised learning (SSL), especially for general audio and music representation learning. While recent methods have demonstrated strong performance, the role of the predictor module used at the output of such SSL systems remains mainly overlooked, despite being crucial for solving the pretext task at hand. In particular, this module should be able to deal with the ambiguity inherent in audio content, especially when it is composed of multiple sound sources. This work proposes a novel enhancement: integrating Multiple Choice Learning (MCL) to explicitly model prediction ambiguity and improve representation quality. We build on top of the recently proposed MATPAC system, improving its prediction and unsupervised classification pretext tasks with MCL. We extensively evaluate our method, MATPAC++, through both linear probing across multiple downstream tasks and fine-tuning on AudioSet, employing a unified protocol that enables rigorous and fair comparisons with state-of-the-art SSL approaches. Results show that our proposal achieves state-of-the-art when fine-tuned on AudioSet and overall state-of-the-art scores on downstream tasks. Additionally, we examine domain specialisation by training exclusively on music data, where our model achieves state-of-the-art performance with significantly improved efficiency.

Citations

• Masked Latent Prediction and Classification for Self-Supervised Audio Representation Learning
• Multiple Choice Learning for Efficient Speech Separation with Many Speakers
• Moshi: a speech-text foundation model for real-time dialogue
• Annealed Multiple Choice Learning: Overcoming limitations of Winner-takes-all with annealing
• GAMA: A Large Audio-Language Model with Advanced Audio Understanding and Complex Reasoning Abilities
• Scaling up masked audio encoder learning for general audio classification
• Masked Modeling Duo: Towards a Universal Audio Pre-training Framework
• Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models
• LLark: A Multimodal Instruction-Following Language Model for Music
• Self-supervised Audio Teacher-Student Transformer for Both Clip-level and Frame-level Tasks
• MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Training
• Listen, Think, and Understand
• DINOv2: Learning Robust Visual Features without Supervision
• BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
• Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture
• BEATs: Audio Pre-Training with Acoustic Tokenizers
• Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation
• Masked Modeling Duo: Learning Representations by Encouraging Both Networks to Model the Input
• Supervised and Unsupervised Learning of Audio Representations for Music Understanding
• Flamingo: a Visual Language Model for Few-Shot Learning
• MAE-AST: Masked Autoencoding Audio Spectrogram Transformer
• Audio Self-supervised Learning: A Survey
• data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language
• HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detection
• iBOT: Image BERT Pre-Training with Online Tokenizer
• LoRA: Low-Rank Adaptation of Large Language Models
• HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units
• Emerging Properties in Self-Supervised Vision Transformers
• Emerging Properties in Self-Supervised Vision Transformers
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• FSD50K: An Open Dataset of Human-Labeled Sound Events
• DMCL: Distillation Multiple Choice Learning for Multimodal Action Recognition
• Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders
• Learning in an Uncertain World: Representing Ambiguity Through Multiple\n Hypotheses
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Stochastic Multiple Choice Learning for Training Diverse Deep Ensembles

Discussions

• I released MATPAC++, an audio SSL foundation model with SOTA performances. 🥇 It solves uncertainty in Masked Audio Modelling by selecting the best prediction among several hypotheses. 🧠 📝 Paper: ar [bsky, 3 points, 0 comments]

Related

• Masked Latent Prediction and Classification for Self-Supervised Audio Representation Learning
• Masked Modeling Duo: Towards a Universal Audio Pre-training Framework
• BEATs: Audio Pre-Training with Acoustic Tokenizers
• Investigating Design Choices in Joint-Embedding Predictive Architectures for General Audio Representation Learning
• Learning music audio representations via weak language supervision