animal2vec and MeerKAT: A self-supervised transformer for rare-event raw audio input and a large-scale reference dataset for bioacoustics

2024/06/03 by Julian C. Schäfer-Zimmermann, Vlad Demartsev, Schäfer-Zimmermann, Julian C. +26 · 3 voices · 3 citations
Biochemistry, Genetics and Molecular Biology · Computer Science · Engineering · Environmental Science · Mathematics · #Animal Vocal Communication and Behavior #Marine animal studies overview #Speech and Audio Processing #cs.AI #cs.SD #eess.AS #q-bio.QM #stat.AP

paper · pdf · doi:10.1111/2041-210x.70218

arxiv published 2024/06/03 · arxiv updated 2026/04/14

Abstract

Bioacoustic research, vital for understanding animal behavior, conservation, and ecology, faces a monumental challenge: analyzing vast datasets where animal vocalizations are rare. While deep learning techniques are becoming standard, adapting them to bioacoustics remains difficult. We address this with animal2vec, an interpretable large transformer model, and a self-supervised training scheme tailored for sparse and unbalanced bioacoustic data. It learns from unlabeled audio and then refines its understanding with labeled data. Furthermore, we introduce and publicly release MeerKAT: Meerkat Kalahari Audio Transcripts, a dataset of meerkat (Suricata suricatta) vocalizations with millisecond-resolution annotations, the largest labeled dataset on non-human terrestrial mammals currently available. Our model outperforms existing methods on MeerKAT and the publicly available NIPS4Bplus birdsong dataset. Moreover, animal2vec performs well even with limited labeled data (few-shot learning). animal2vec and MeerKAT provide a new reference point for bioacoustic research, enabling scientists to analyze large amounts of data even with scarce ground truth information.

Citations

• Terrestrial Passive Acoustic Monitoring: Review and Perspectives
• The iNaturalist Sounds Dataset
• Can Masked Autoencoders Also Listen to Birds?
• NatureLM-audio: an Audio-Language Foundation Model for Bioacoustics
• BirdSet: A Large-Scale Dataset for Audio Classification in Avian Bioacoustics
• Transferable Models for Bioacoustics with Human Language Supervision
• A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicity
• Efficient Self-supervised Learning with Contextualized Target Representations for Vision, Speech and Language
• Robust Speech Recognition via Large-Scale Weak Supervision
• AVES: Animal Vocalization Encoder based on Self-Supervision
• On the Importance and Applicability of Pre-Training for Federated Learning
• Is BERT Robust to Label Noise? A Study on Learning with Noisy Labels in Text Classification
• data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language
• Pushing the limits of self-supervised ResNets: Can we outperform supervised learning without labels on ImageNet?
• W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training
• Proposal-based Few-shot Sound Event Detection for Speech and Environmental Sounds with Perceivers
• Can contrastive learning avoid shortcut solutions?
• HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units
• A Survey of Transformers
• Emerging Properties in Self-Supervised Vision Transformers
• AST: Audio Spectrogram Transformer
• BYOL for Audio: Self-Supervised Learning for General-Purpose Audio Representation
• Transformers in Vision: A Survey
• A Survey on Contrastive Self-supervised Learning
• Limitations of Autoregressive Models and Their Alternatives
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Contrastive Learning of General-Purpose Audio Representations
• Learning from Noisy Labels with Deep Neural Networks: A Survey
• wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
• Big Self-Supervised Models are Strong Semi-Supervised Learners
• Bootstrap your own latent: A new approach to self-supervised Learning
• Language Models are Few-Shot Learners
• Improved Baselines with Momentum Contrastive Learning
• A Simple Framework for Contrastive Learning of Visual Representations
• Momentum Contrast for Unsupervised Visual Representation Learning
• RoBERTa: A Robustly Optimized BERT Pretraining Approach
• NIPS4Bplus: a richly annotated birdsong audio dataset
• Speaker Recognition from Raw Waveform with SincNet
• Data-Efficient Weakly Supervised Learning for Low-Resource Audio Event Detection Using Deep Learning
• Representation Learning with Contrastive Predictive Coding
• Learning from Between-class Examples for Deep Sound Recognition
• Densely Connected Convolutional Networks
• Deep Residual Learning for Image Recognition
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• The Orchive : Data mining a massive bioacoustic archive
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Cited by

• Explainability of CNN Based Classification Models for Acoustic Signal
• What Matters for Bioacoustic Encoding
• Foundation Models for Bioacoustics -- a Comparative Review

Discussions

• "animal2vec and MeerKAT: A self-supervised transformer for rare-event raw audio input and a large-scale reference dataset for bioacoustics" recently published! http://doi.org/10.1111/2041-210x.70218 # [bsky, 9 points, 0 comments]
• I <3 folks who share code and data. Schäfer-Zimmermann & team: animal2vec and MeerKAT: A self-supervised transformer for rare-event raw audio input and a large-scale reference dataset for bioacoustics [bsky, 3 points, 0 comments]
• "animal2vec and MeerKAT: A self-supervised transformer for rare-event raw audio input and a large-scale reference dataset for bioacoustics" https://arxiv.org/abs/2406.01253 #bioacoustics #machinelearn [bsky, 0 points, 0 comments]

Related

• AVES: Animal Vocalization Encoder based on Self-Supervision
• NatureLM-audio: an Audio-Language Foundation Model for Bioacoustics
• Application of machine learning in analyses of bioacoustics data from terrestrial mammals and insects: a systematic review and appraisal of methods.
• Transferable Models for Bioacoustics with Human Language Supervision
• BEANS: The Benchmark of Animal Sounds