2024/11/26 by Hoyoung Choi, Ho-young Choi, Seungwan Jin +1 · 1 citation
Biochemistry, Genetics and Molecular Biology · Engineering · #CCD and CMOS Imaging Sensors #Cell Image Analysis Techniques #Image Processing Techniques and Applications
paper · pdf · doi:10.1007/s11263-024-02290-6
openalex publication_date 2024/11/26 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/01
Vision transformers use [CLS] token to predict image classes. Their explainability visualization has been studied using relevant information from the [CLS] token or focusing on attention scores during self-attention. However, such visualization is challenging because of the dependence of the interpretability of a vision transformer on skip connections and attention operators, the instability of non-linearities in the learning process, and the limited reflection of self-attention scores on relevance. We argue that the output patch embeddings in a vision transformer preserve the image information of each patch location, which can facilitate the prediction of an image class. In this paper, we propose ICEv2 (ICEv2: \underline\varvecI nterpretability, \underline\varvecC omprehensiveness, and \underline\varvecE xplainability in Vision Transformer), an explainability visualization method that addresses the limitations of ICE (i.e., high dependence of hyperparameters on performance and the inability to preserve the model’s properties) by minimizing the number of training encoder layers, redesigning the MLP layer, and optimizing hyperparameters along with various model size. Overall, ICEv2 shows higher efficiency, performance, robustness, and scalability than ICE. On the ImageNet-Segmentation dataset, ICEv2 outperformed all explainability visualization methods in all cases depending on the model size. On the Pascal VOC dataset, ICEv2 outperformed both self-supervised and supervised methods on Jaccard similarity. In the unsupervised single object discovery, where untrained classes are present in the images, ICEv2 effectively distinguished between foreground and background, showing performance comparable to the previous state-of-the-art. Lastly, ICEv2 can be trained with significantly lower training computational complexity.