Input-independent Attention Weights Are Expressive Enough: A Study of Attention in Self-supervised Audio Transformers

2020/06/09 by Tsung-Han Wu, Wu, Tsung-Han, Chun-Chen Hsieh +7
Computer Science · #Speech Recognition and Synthesis #Music and Audio Processing #Speech and Audio Processing

paper · pdf · doi:10.48550/arxiv.2006.05174

Abstract

In this paper, we seek solutions for reducing the computation complexity of transformer-based models for speech representation learning. We evaluate 10 attention algorithms; then, we pre-train the transformer-based model with those attention algorithms in a self-supervised fashion and treat them as feature extractors on downstream tasks, including phoneme classification and speaker classification. With the assistance of t-SNE, PCA and some observation, the attention weights in self-supervised audio transformers can be categorized into four general cases. Based on these cases and some analyses, we are able to use a specific set of attention weights to initialize the model. Our approach shows comparable performance to the typical self-attention yet requires 20% less time in both training and inference.

Related

• Attention or Convolution: Transformer Encoders in Audio Language Models for Inference Efficiency
• On the Usefulness of Self-Attention for Automatic Speech Recognition with Transformers
• When to Use Efficient Self Attention? Profiling Text, Speech and Image Transformer Variants
• Guiding Attention for Self-Supervised Learning with Transformers
• The silence of the weights: an investigation of structural pruning strategies for attention-based audio signal architectures