Unmute the Patch Tokens: Rethinking Probing in Multi-Label Audio Classification

2025/09/29 by Lukas Rauch, René Heinrich, Rauch, Lukas +11 · 2 citations
Arts and Humanities · Computer Science · Engineering · #Diverse Musicological Studies #FOS: Computer and information sciences #Machine Learning (cs.LG) #Music and Audio Processing #Sound (cs.SD) #Water Systems and Optimization

paper · pdf · doi:10.48550/arxiv.2509.24901

openalex publication_date 2025/09/29 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Although probing frozen models has become a standard evaluation paradigm, self-supervised learning in audio defaults to fine-tuning when pursuing state-of-the-art on AudioSet. A key reason is that global pooling creates an information bottleneck causing linear probes to misrepresent the embedding quality: The cls-token discards crucial token information about dispersed, localized events in audio. This weakness is rooted in the mismatch between the pretraining objective (globally) and the downstream task (localized). Across a comprehensive benchmark of 13 datasets and 6 spectrogram-based encoders, we investigate the global pooling bottleneck. We introduce binarized prototypical probes: a lightweight and simple pooling method that learns prototypes to perform class-wise information aggregation. Despite its simplicity, our method notably outperforms linear and attentive probing. Our work establishes probing as a competitive and efficient paradigm for evaluating audio SSL models, challenging the reliance on costly fine-tuning.

Citations

• SSLAM: Enhancing Self-Supervised Models with Audio Mixtures for Polyphonic Soundscapes
• Can Masked Autoencoders Also Listen to Birds?
• M2D-CLAP: Exploring General-purpose Audio-Language Representations Beyond CLAP
• Cluster and Predict Latent Patches for Improved Masked Image Modeling
• Towards Deep Active Learning in Avian Bioacoustics
• Scaling up masked audio encoder learning for general audio classification
• AudioProtoPNet: An interpretable deep learning model for bird sound classification
• BirdSet: A Large-Scale Dataset for Audio Classification in Avian Bioacoustics
• MIM-Refiner: A Contrastive Learning Boost from Intermediate Pre-Trained Representations
• Revisiting Feature Prediction for Learning Visual Representations from Video
• Scalable Pre-training of Large Autoregressive Image Models
• EAT: Self-Supervised Pre-Training with Efficient Audio Transformer
• EnCodecMAE: Leveraging neural codecs for universal audio representation learning
• Self-supervised Audio Teacher-Student Transformer for Both Clip-level and Frame-level Tasks
• Masked Autoencoders with Multi-Window Local-Global Attention Are Better Audio Learners
• What Do Self-Supervised Vision Transformers Learn?
• DINOv2: Learning Robust Visual Features without Supervision
• Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture
• BEATs: Audio Pre-Training with Acoustic Tokenizers
• Masked Autoencoders that Listen
• HEAR: Holistic Evaluation of Audio Representations
• Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution
• Context Autoencoder for Self-Supervised Representation Learning
• data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language
• Masked Autoencoders Are Scalable Vision Learners
• BEiT: BERT Pre-Training of Image Transformers
• Emerging Properties in Self-Supervised Vision Transformers
• Emerging Properties in Self-Supervised Vision Transformers
• DESED-FL and URBAN-FL: Federated Learning Datasets for Sound Event Detection
• FSD50K: An Open Dataset of Human-Labeled Sound Events
• Asymmetric Loss For Multi-Label Classification
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
• A Simple Framework for Contrastive Learning of Visual Representations
• Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition
• Attention-based Deep Multiple Instance Learning
• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• Attention, Please! Revisiting Attentive Probing Through the Lens of Efficiency

Cited by

• Cleaning the Pool: Progressive Filtering of Unlabeled Pools in Deep Active Learning
• Data-Efficient Self-Supervised Algorithms for Fine-Grained Birdsong Analysis

Related

• BEATs: Audio Pre-Training with Acoustic Tokenizers
• UNFUSED: UNsupervised Finetuning Using SElf supervised Distillation
• How Should We Extract Discrete Audio Tokens from Self-Supervised Models?
• Audio Contrastive-based Fine-tuning: Decoupling Representation Learning and Classification
• SSLAM: Enhancing Self-Supervised Models with Audio Mixtures for Polyphonic Soundscapes