HRFormer: High-Resolution Transformer for Dense Prediction

2021/10/18 by Yuan, Yuhui, Fu, Rao, Huang, Lang +4 · 5 citations
#Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2110.09408

Abstract

We present a High-Resolution Transformer (HRFormer) that learns high-resolution representations for dense prediction tasks, in contrast to the original Vision Transformer that produces low-resolution representations and has high memory and computational cost. We take advantage of the multi-resolution parallel design introduced in high-resolution convolutional networks (HRNet), along with local-window self-attention that performs self-attention over small non-overlapping image windows, for improving the memory and computation efficiency. In addition, we introduce a convolution into the FFN to exchange information across the disconnected image windows. We demonstrate the effectiveness of the High-Resolution Transformer on both human pose estimation and semantic segmentation tasks, e.g., HRFormer outperforms Swin transformer by 1.3 AP on COCO pose estimation with 50% fewer parameters and 30% fewer FLOPs. Code is available at: https://github.com/HRNet/HRFormer.

Cited by

• Accelerating Vision Transformers with Adaptive Patch Sizes
• SDPose: Exploiting Diffusion Priors for Out-of-Domain and Robust Pose Estimation
• OmniSegmentor: A Flexible Multi-Modal Learning Framework for Semantic Segmentation
• Parse Graph-Based Visual-Language Interaction for Human Pose Estimation
• Dynamic Pattern Alignment Learning for Pretraining Lightweight Human-Centric Vision Models

Related

• Vision Transformers for Dense Prediction
• HR-NAS: Searching Efficient High-Resolution Neural Architectures with Lightweight Transformers
• Deep High-Resolution Representation Learning for Visual Recognition
• Deep High-Resolution Representation Learning for Visual Recognition
• Expediting Large-Scale Vision Transformer for Dense Prediction without Fine-tuning