Multi-Scale Self-Attention for Text Classification

2019/12/02 by Guo, Qipeng, Qiu, Xipeng, Liu, Pengfei +2 · 1 citation
#Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.1912.00544

Abstract

In this paper, we introduce the prior knowledge, multi-scale structure, into self-attention modules. We propose a Multi-Scale Transformer which uses multi-scale multi-head self-attention to capture features from different scales. Based on the linguistic perspective and the analysis of pre-trained Transformer (BERT) on a huge corpus, we further design a strategy to control the scale distribution for each layer. Results of three different kinds of tasks (21 datasets) show our Multi-Scale Transformer outperforms the standard Transformer consistently and significantly on small and moderate size datasets.

Cited by

• MSVIT: Improving Spiking Vision Transformer Using Multi-scale Attention Fusion

Related

• Multi-Level Attention and Contrastive Learning for Enhanced Text Classification with an Optimized Transformer
• Learning Multiscale Transformer Models for Sequence Generation
• Low Rank Factorization for Compact Multi-Head Self-Attention
• MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
• When to Use Efficient Self Attention? Profiling Text, Speech and Image Transformer Variants