2024/03/15 by Sanghyeok Lee, Lee, Sanghyeok, Joonmyung Choi +3 · 7 citations
Computer Science · Engineering · #Computer Vision and Pattern Recognition (cs.CV) #Currency Recognition and Detection #FOS: Computer and information sciences #Industrial Vision Systems and Defect Detection #Infrared Target Detection Methodologies
paper · pdf · doi:10.48550/arxiv.2403.10030
openalex publication_date 2024/03/15 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Vision Transformer (ViT) has emerged as a prominent backbone for computer vision. For more efficient ViTs, recent works lessen the quadratic cost of the self-attention layer by pruning or fusing the redundant tokens. However, these works faced the speed-accuracy trade-off caused by the loss of information. Here, we argue that token fusion needs to consider diverse relations between tokens to minimize information loss. In this paper, we propose a Multi-criteria Token Fusion (MCTF), that gradually fuses the tokens based on multi-criteria (e.g., similarity, informativeness, and size of fused tokens). Further, we utilize the one-step-ahead attention, which is the improved approach to capture the informativeness of the tokens. By training the model equipped with MCTF using a token reduction consistency, we achieve the best speed-accuracy trade-off in the image classification (ImageNet1K). Experimental results prove that MCTF consistently surpasses the previous reduction methods with and without training. Specifically, DeiT-T and DeiT-S with MCTF reduce FLOPs by about 44% while improving the performance (+0.5%, and +0.3%) over the base model, respectively. We also demonstrate the applicability of MCTF in various Vision Transformers (e.g., T2T-ViT, LV-ViT), achieving at least 31% speedup without performance degradation. Code is available at https://github.com/mlvlab/MCTF.