A Whisper transformer for audio captioning trained with synthetic captions and transfer learning

2023/05/15 by Kadlčík, Marek, Hájek, Adam, Kieslich, Jürgen +1 · 2 citations
#Audio and Speech Processing (eess.AS) #FOS: Computer and information sciences #FOS: Electrical engineering #Machine Learning (cs.LG) #Sound (cs.SD) #electronic engineering #information engineering

paper · doi:10.48550/arxiv.2305.09690

Abstract

The field of audio captioning has seen significant advancements in recent years, driven by the availability of large-scale audio datasets and advancements in deep learning techniques. In this technical report, we present our approach to audio captioning, focusing on the use of a pretrained speech-to-text Whisper model and pretraining on synthetic captions. We discuss our training procedures and present our experiments' results, which include model size variations, dataset mixtures, and other hyperparameters. Our findings demonstrate the impact of different training strategies on the performance of the audio captioning model. Our code and trained models are publicly available on GitHub and Hugging Face Hub.

Cited by

• Technical Report of Nomi Team in the Environmental Sound Deepfake Detection Challenge 2026
• AUGUSTUS: An LLM-Driven Multimodal Agent System with Contextualized User Memory

Related

• Audio Captioning using Pre-Trained Large-Scale Language Model Guided by Audio-based Similar Caption Retrieval
• Improving Text-To-Audio Models with Synthetic Captions
• Training Audio Captioning Models without Audio
• Investigations in Audio Captioning: Addressing Vocabulary Imbalance and Evaluating Suitability of Language-Centric Performance Metrics
• Efficient Audio Captioning Transformer with Patchout and Text Guidance