PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition
2019/12/21 by Qiuqiang Kong, Kong, Qiuqiang, Yin Cao +9 · 161 citations
Computer Science · Engineering · #Audio and Speech Processing (eess.AS) #FOS: Computer and information sciences #FOS: Electrical engineering #Music and Audio Processing #Sound (cs.SD) #Speech Recognition and Synthesis #Speech and Audio Processing #cs.SD #eess.AS #electronic engineering #information engineering
paper · pdf · doi:10.48550/arxiv.1912.10211
14 pages
openalex publication_date 2019/12/21 · openalex created_date 2019/12/26 · arxiv created 2020/08/23 · arxiv updated 2020/08/25 · openalex updated_date 2026/07/28
Abstract
Audio pattern recognition is an important research topic in the machine learning area, and includes several tasks such as audio tagging, acoustic scene classification, music classification, speech emotion classification and sound event detection. Recently, neural networks have been applied to tackle audio pattern recognition problems. However, previous systems are built on specific datasets with limited durations. Recently, in computer vision and natural language processing, systems pretrained on large-scale datasets have generalized well to several tasks. However, there is limited research on pretraining systems on large-scale datasets for audio pattern recognition. In this paper, we propose pretrained audio neural networks (PANNs) trained on the large-scale AudioSet dataset. These PANNs are transferred to other audio related tasks. We investigate the performance and computational complexity of PANNs modeled by a variety of convolutional neural networks. We propose an architecture called Wavegram-Logmel-CNN using both log-mel spectrogram and waveform as input feature. Our best PANN system achieves a state-of-the-art mean average precision (mAP) of 0.439 on AudioSet tagging, outperforming the best previous system of 0.392. We transfer PANNs to six audio pattern recognition tasks, and demonstrate state-of-the-art performance in several of those tasks. We have released the source code and pretrained models of PANNs: https://github.com/qiuqiangkong/audiosettaggingcnn.
Citations
Cited by
- AudioGAN: A Compact and Efficient Framework for Real-Time High-Fidelity Text-to-Audio Generation
- Identifying Dolphin Whistle Producers With Deep Learning: Moving Beyond Signature Whistles
- Zero-Shot to Zero-Lies: Detecting Bengali Deepfake Audio through Transfer Learning
- Do Foundational Audio Encoders Understand Music Structure?
- JoVA: Unified Multimodal Learning for Joint Video-Audio Generation and Editing
- ASK: Adaptive Self-improving Knowledge Framework for Audio Text Retrieval
- PAVAS: Physics-Aware Video-to-Audio Synthesis
- DreamFoley: Scalable VLMs for High-Fidelity Video-to-Audio Generation
- Agent-Based Modular Learning for Multimodal Emotion Recognition in Human-Agent Systems
- Hear What Matters! Text-conditioned Selective Video-to-Audio Generation
- Fiber Bundle Networks: A Geometric Machine Learning Paradigm
- CACARA: Cross-Modal Alignment Leveraging a Text-Centric Approach for Cost-Effective Multimodal and Multilingual Learning
- See, Rank, and Filter: Important Word-Aware Clip Filtering via Scene Understanding for Moment Retrieval and Highlight Detection
- Multimodal Real-Time Anomaly Detection and Industrial Applications
- DHAuDS: A Dynamic and Heterogeneous Audio Benchmark for Test-Time Adaptation
- Revisiting Audio-language Pretraining for Learning General-purpose Audio Representation
- SceneGuard: Training-Time Voice Protection with Scene-Consistent Audible Background Noise
- Fine-tuning Pre-trained Audio Models for COVID-19 Detection: A Technical Report
- Reconstruction-Driven Multimodal Representation Learning for Automated Media Understanding
- FoleyBench: A Benchmark For Video-to-Audio Models
- AudioNet: Supervised Deep Hashing for Retrieval of Similar Audio Events
- SAR-LM: Symbolic Audio Reasoning with Large Language Models
- PromptSep: Generative Audio Separation via Multimodal Prompting
- Ripple: Real-Time Streaming Audio-Video Generation With Cross-Modal Recurrent Memory
- Joint Analysis of Acoustic Scenes and Sound Events Based on Semi-Supervised Training of Sound Events With Partial Labels
- Model-Guided Dual-Role Alignment for High-Fidelity Open-Domain Video-to-Audio Generation
- SAO-Instruct: Free-form Audio Editing using Natural Language Instructions
- Foley Control: Aligning a Frozen Latent Text-to-Audio Model to Video
- Compressing Quaternion Convolutional Neural Networks for Audio Classification
- Elementary, My Dear Watson: Non-Invasive Neural Keyword Spotting in the LibriBrain Dataset
- GuitarFlow: Realistic Electric Guitar Synthesis From Tablatures via Flow Matching and Style Transfer
- Not in Sync: Unveiling Temporal Bias in Audio Chat Models
- UALM: Unified Audio Language Model for Understanding, Generation and Reasoning
- Go witheFlow: Real-time Emotion Driven Audio Effects Modulation
- Bias beyond Borders: Global Inequalities in AI-Generated Music
- MelTok: 2D Tokenization for Single-Codebook Audio Compression
- MMAudioSep: Taming Video-to-Audio Generative Model Towards Video/Text-Queried Sound Separation
- Detecting and Mitigating Insertion Hallucination in Video-to-Audio Generation
- Sci-Phi: A Large Language Model Spatial Audio Descriptor
- Language Model Based Text-to-Audio Generation: Anti-Causally Aligned Collaborative Residual Transformers
- SALSA-V: Shortcut-Augmented Long-form Synchronized Audio from Videos
- Learning Domain-Robust Bioacoustic Representations for Mosquito Species Classification with Contrastive Learning and Distribution Alignment
- Video Object Segmentation-Aware Audio Generation
- Contrastive Diffusion Guidance for Spatial Inverse Problems
- Ovi: Twin Backbone Cross-Modal Fusion for Audio-Video Generation
- EMO-TTA: Improving Test-Time Adaptation of Audio-Language Models for Speech Emotion Recognition
- UniFlow-Audio: Unified Flow Matching for Audio Generation from Omni-Modalities
- AudioMoG: Guiding Audio Generation with Mixture-of-Guidance
- Cross-Dialect Bird Species Recognition with Dialect-Calibrated Augmentation
- MusicWeaver: Programmable Long-Form Music Generation with Provably Local Editing
- MultiSoundGen: Video-to-Audio Generation for Multi-Event Scenarios via SlowFast Contrastive Audio-Visual Pretraining and Direct Preference Optimization
- An overview of neural architectures for self-supervised audio representation learning from masked spectrograms
- Scattering Transformer: A Training-Free Transformer Architecture for Heart Murmur Detection
- StereoFoley: Object-Aware Stereo Audio Generation from Video
- A Survey on Evaluation Metrics for Music Generation
- The Rhythm In Anything: Audio-Prompted Drums Generation with Masked Language Modeling
- Pre-training Autoencoder for Acoustic Event Classification via Blinky
- Aligning Audio Captions with Human Preferences
- Comprehensive Evaluation of CNN-Based Audio Tagging Models on Resource-Constrained Devices
- RFM-Editing: Rectified Flow Matching for Text-guided Audio Editing
- Noise Supervised Contrastive Learning and Feature-Perturbed for Anomalous Sound Detection
- MAGIC-Enhanced Keyword Prompting for Zero-Shot Audio Captioning with CLIP Models
- Can Large Audio Language Models Understand Audio Well? Speech, Scene and Events Understanding Benchmark for LALMs
- Improving Audio Event Recognition with Consistency Regularization
- Region-Specific Audio Tagging for Spatial Sound
- HunyuanVideo-Foley: Multimodal Diffusion with Representation Alignment for High-Fidelity Foley Audio Generation
- D-CAT: Decoupled Cross-Attention Transfer between Sensor Modalities for Unimodal Inference
- MeanFlow-Accelerated Multimodal Video-to-Audio Synthesis via One-Step Generation
- Continuous Audio Language Models
- UniVerse-1: Unified Audio-Video Generation via Stitching of Experts
- DreamAudio: Customized Text-to-Audio Generation with Diffusion Models
- Efficient Video-to-Audio Generation via Multiple Foundation Models Mapper
- WatchHAR: Real-time On-device Human Activity Recognition System for Smartwatches
- Training a Perceptual Model for Evaluating Auditory Similarity in Music Adversarial Attack
- Online incremental learning for audio classification using a pretrained audio model
- AudioSet-R: A Refined AudioSet with Multi-Stage LLM Label Reannotation
- ECHO: Frequency-aware Hierarchical Encoding for Variable-length Signals
- Multimodal Data Storage and Retrieval for Embodied AI: A Survey
- FoleySpace: Vision-Aligned Binaural Spatial Audio Generation
- MuSACo: Multimodal Subject-Specific Selection and Adaptation for Expression Recognition with Co-Training
- Soundscape-based evaluation of small-scale forest management interventions
- Auditory Intelligence: Understanding the World Through Sound
- eMotions: A Large-Scale Dataset and Audio-Visual Fusion Network for Emotion Analysis in Short-form Videos
- Acoustic Non-Stationarity Objective Assessment with Hard Label Criteria for Supervised Learning Models
- MeanAudio: Fast and Faithful Text-to-Audio Generation with Mean Flows
- Test-Time Adaptation for Video Highlight Detection Using Meta-Auxiliary Learning and Cross-Modality Hallucinations
- Audio Does Matter: Importance-Aware Multi-Granularity Fusion for Video Moment Retrieval
- NVSpeech: An Integrated and Scalable Pipeline for Human-Like Speech Modeling with Paralinguistic Vocalizations
- Charting 15 years of progress in deep learning for speech emotion recognition: A replication study
- AudioGen-Omni: A Unified Multimodal Diffusion Transformer for Video-Synchronized Audio, Speech, and Song Generation
- Evaluating and Improving the Robustness of Speech Command Recognition Models to Noise and Distribution Shifts
- From Waveforms to Pixels: A Survey on Audio-Visual Segmentation
- Controllable Video-to-Music Generation with Multiple Time-Varying Conditions
- Improving Deep Learning-based Respiratory Sound Analysis with Frequency Selection and Attention Mechanism
- ChoreoMuse: Robust Music-to-Dance Video Generation with Style Transfer and Beat-Adherent Motion
- The ISLab Solution to the Algonauts Challenge 2025: A Multimodal Deep Learning Approach to Brain Response Prediction
- TENET: A Time-reversal Enhancement Network for Noise-robust ASR
- MelT: A Portable, Single-GEMM Mel Audio Frontend via Non-Uniform DFT with Measured Latency and Energy Gains on GPUs
- DualDub: Video-to-Soundtrack Generation via Joint Speech and Background Audio Synthesis
- Enhancing Stereo Sound Event Detection with BiMamba and Pretrained PSELDnet
- Enforcing Speech Content Privacy in Environmental Sound Recordings using Segment-wise Waveform Reversal
- IMPACT: Industrial Machine Perception via Acoustic Cognitive Transformer
- Differentiable Reward Optimization for LLM based TTS system
- EXPOTION: Facial Expression and Motion Control for Multimodal Music Generation
- Hear-Your-Click: Interactive Object-Specific Video-to-Audio Generation
- CLEP-DG: Contrastive Learning for Speech Emotion Domain Generalization via Soft Prompt Tuning
- Fx-Encoder++: Extracting Instrument-Wise Audio Effects Representations from Mixtures
- Real-Time Emergency Vehicle Siren Detection with Efficient CNNs on Embedded Hardware
- From Large-scale Audio Tagging to Real-Time Explainable Emergency Vehicle Sirens Detection
- DeepOmni: Towards Seamless and Smart Speech Interaction with Adaptive Modality-Specific MoE
- Exploring Adapter Design Tradeoffs for Low Resource Music Generation
- Step-by-Step Video-to-Audio Synthesis via Negative Audio Guidance
- ThinkSound: Chain-of-Thought Reasoning in Multimodal Large Language Models for Audio Generation and Editing
- Kling-Foley: Multimodal Diffusion Transformer for High-Quality Video-to-Audio Generation
- Benchmarking Music Generation Models and Metrics via Human Preference Studies
- MuseControlLite: Multifunctional Music Generation with Lightweight Conditioners
- A multi-stage augmented multimodal interaction network for fish feeding intensity quantification
- Stereo sound event localization and detection based on PSELDnet pretraining and BiMamba sequence modeling
- Video-Guided Text-to-Music Generation Using Public Domain Movie Collections
- SSLAM: Enhancing Self-Supervised Models with Audio Mixtures for Polyphonic Soundscapes
- Survey on the Evaluation of Generative Models in Music
- Sounding that Object: Interactive Object-Aware Image to Audio Generation
- Cross-Modal Urban Sensing: Evaluating Sound-Vision Alignment Across Street-Level and Aerial Imagery
- Enhancing Speech Emotion Recognition with Graph-Based Multimodal Fusion and Prosodic Features for the Speech Emotion Recognition in Naturalistic Conditions Challenge at Interspeech 2025
- In-the-wild Audio Spatialization with Flexible Text-guided Localization
- Improving Respiratory Sound Classification with Architecture-Agnostic Knowledge Distillation from Ensembles
- AudioTurbo: Fast Text-to-Audio Generation with Rectified Diffusion
- Patient-Aware Feature Alignment for Robust Lung Sound Classification:Cohesion-Separation and Global Alignment Losses
- Hybrid Disagreement-Diversity Active Learning for Bioacoustic Sound Event Detection
- Conditional Diffusion Models with Classifier-Free Gibbs-like Guidance
- Rhapsody: A Dataset for Highlight Detection in Podcasts
- Large-Scale MIDI-based Composer Classification
- Self-supervised learning method using multiple sampling strategies for general-purpose audio representation
- Learning Normal Patterns in Musical Loops
- SpecMaskFoley: Steering Pretrained Spectral Masked Generative Transformer Toward Synchronized Video-to-audio Synthesis via ControlNet
- Uncertainty-Aware Crossmodal Fusion for Classification of Animal Behavior
- Can Foundation Models Hear What Made That Sound? A Tiered Benchmark of Audio-Language Models and Traditional Classifiers for Closed-Set Sound Source Identification
- Towards Pre-training an Effective Respiratory Audio Foundation Model
- Discrete Audio Representations for Automated Audio Captioning
- Scene2Sound: Auditory-Grounded Soundscape Generation for 3D Gaussian Worlds
- Revisiting SSL for sound event detection: complementary fusion and adaptive post-processing
- Improving Inference-Time Optimisation for Vocal Effects Style Transfer with a Gaussian Prior
- Bridging Ears and Eyes: Analyzing Audio and Visual Large Language Models to Humans in Visible Sound Recognition and Reducing Their Sensory Gap via Cross-Modal Distillation
- A Survey of Large Audio Language Models: Generalization, Trustworthiness, and Outlook
- Bagpiper: Solving Open-Ended Audio Tasks via Rich Captions
- Impact of transfer learning methods and dataset characteristics on generalization in birdsong classification
- Acoustic Scene Classification with Spectrogram Processing Strategies
- Quality Audio Prototyping: a prototype system for unified sound retrieval and procedural generation
- From Aesthetics to Human Preferences: Comparative Perspectives of Evaluating Text-to-Music Systems
- Unleashing the Power of Natural Audio Featuring Multiple Sound Sources
- Towards Robust Version Identification in the Wild: A Dataset, Benchmark, and Fine-Tuning Study
- Deep Learning for Real-Time Sound Order Recognition in Human-Robot Interaction
- Text-based depression detection on sparse data
- Exploring the User Experience of AI-Assisted Sound Searching Systems for Creative Workflows
- Audio-Visual Class-Incremental Learning for Fish Feeding intensity Assessment in Aquaculture
- Transformation of audio embeddings into interpretable, concept-based representations
- A Survey on Cross-Modal Interaction Between Music and Multimodal Data
- Temporal Attention Pooling for Frequency Dynamic Convolution in Sound Event Detection
- KVAE: Family of Tokenizers for Multimodal Generative Models
- EchoEdit: Stabilizing Inversion-Free Audio Editing via Optimal Transport Geometry
- On feature representations for marmoset vocal communication analysis
- Transformer-based audio-visual multimodal fusion for fine-grained recognition of individual sow nursing behaviour
Related