Efficient Large-scale Audio Tagging via Transformer-to-CNN Knowledge Distillation

2022/11/09 by Florian Schmid, Schmid, Florian, Khaled Koutini +3 · 8 citations
Computer Science · #Audio and Speech Processing (eess.AS) #FOS: Computer and information sciences #FOS: Electrical engineering #Machine Learning (cs.LG) #Music and Audio Processing #Sound (cs.SD) #Speech Recognition and Synthesis #Speech and Audio Processing #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2211.04772

openalex publication_date 2022/11/09 · openalex created_date 2022/11/15 · openalex updated_date 2026/07/28

Abstract

Audio Spectrogram Transformer models rule the field of Audio Tagging, outrunning previously dominating Convolutional Neural Networks (CNNs). Their superiority is based on the ability to scale up and exploit large-scale datasets such as AudioSet. However, Transformers are demanding in terms of model size and computational requirements compared to CNNs. We propose a training procedure for efficient CNNs based on offline Knowledge Distillation (KD) from high-performing yet complex transformers. The proposed training schema and the efficient CNN design based on MobileNetV3 results in models outperforming previous solutions in terms of parameter and computational efficiency and prediction performance. We provide models of different complexity levels, scaling from low-complexity models up to a new state-of-the-art performance of .483 mAP on AudioSet. Source Code available at: https://github.com/fschmid56/EfficientAT

Cited by

• VibOmni: Towards Scalable Bone-conduction Speech Enhancement on Earables
• Compressing Quaternion Convolutional Neural Networks for Audio Classification
• DeepASA: An Object-Oriented Multi-Purpose Network for Auditory Scene Analysis
• Comprehensive Evaluation of CNN-Based Audio Tagging Models on Resource-Constrained Devices
• AMMKD: Adaptive Multimodal Multi-teacher Distillation for Lightweight Vision-Language Models
• TinyMusician: On-Device Music Generation with Knowledge Distillation and Mixed Precision Quantization
• Interpretable Embeddings of Speech Enhance and Explain Brain Encoding Performance of Audio Models
• VAT-KG: Knowledge-Intensive Multimodal Knowledge Graph Dataset for Retrieval-Augmented Generation

Related

• Dynamic Convolutional Neural Networks as Efficient Pre-trained Audio Models
• Efficient Supervised Training of Audio Transformers for Music Representation Learning
• Audio Transformers
• CMKD: CNN/Transformer-Based Cross-Model Knowledge Distillation for Audio\n Classification
• ElasticAST: An Audio Spectrogram Transformer for All Length and Resolutions