HRFormer: High-Resolution Transformer for Dense Prediction

2021/10/18 by Yuhui Yuan, Yuan, Yuhui, Rao Fu +11 · 12 citations
Computer Science · #Advanced Neural Network Applications #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Human Pose and Action Recognition #cs.CV

paper · pdf · doi:10.48550/arxiv.2110.09408

Accepted at NeurIPS 2021

openalex publication_date 2021/10/18 · arxiv created 2021/11/07 · arxiv updated 2021/11/09 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We present a High-Resolution Transformer (HRFormer) that learns high-resolution representations for dense prediction tasks, in contrast to the original Vision Transformer that produces low-resolution representations and has high memory and computational cost. We take advantage of the multi-resolution parallel design introduced in high-resolution convolutional networks (HRNet), along with local-window self-attention that performs self-attention over small non-overlapping image windows, for improving the memory and computation efficiency. In addition, we introduce a convolution into the FFN to exchange information across the disconnected image windows. We demonstrate the effectiveness of the High-Resolution Transformer on both human pose estimation and semantic segmentation tasks, e.g., HRFormer outperforms Swin transformer by 1.3 AP on COCO pose estimation with 50% fewer parameters and 30% fewer FLOPs. Code is available at: https://github.com/HRNet/HRFormer.

Citations

• DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs
• Rethinking Atrous Convolution for Semantic Image Segmentation
• mixup: Beyond Empirical Risk Minimization
• OCNet: Object Context Network for Scene Parsing
• Decoupled Weight Decay Regularization
• Stand-Alone Self-Attention in Vision Models
• Interlaced Sparse Self-Attention for Semantic Segmentation
• Deep High-Resolution Representation Learning for Visual Recognition
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet
• Is Space-Time Attention All You Need for Video Understanding?
• Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions
• Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
• CvT: Introducing Convolutions to Vision Transformers
• CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification
• Multiscale Vision Transformers
• LocalViT: Analyzing Locality in Vision Transformers
• Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers
• CoAtNet: Marrying Convolution and Attention for All Data Sizes

Cited by

• Accelerating Vision Transformers with Adaptive Patch Sizes
• SDPose: Exploiting Diffusion Priors for Out-of-Domain and Robust Pose Estimation
• OmniSegmentor: A Flexible Multi-Modal Learning Framework for Semantic Segmentation
• Parse Graph-Based Visual-Language Interaction for Human Pose Estimation
• Dynamic Pattern Alignment Learning for Pretraining Lightweight Human-Centric Vision Models
• Spatial Frequency Modulation for Semantic Segmentation
• KptLLM++: Towards Generic Keypoint Comprehension with Large Language Model
• FreeDNA: Endowing Domain Adaptation of Diffusion-Based Dense Prediction with Training-Free Domain Noise Alignment
• RoadFormer : Local-Global Feature Fusion for Road Surface Classification in Autonomous Driving
• PoseBH: Prototypical Multi-Dataset Training Beyond Human Pose Estimation
• A Novel Hybrid Approach for Retinal Vessel Segmentation with Dynamic Long-Range Dependency and Multi-Scale Retinal Edge Fusion Enhancement
• SRVP: Strong Recollection Video Prediction Model Using Attention-Based Spatiotemporal Correlation Fusion

Related

• Vision Transformers for Dense Prediction
• HR-NAS: Searching Efficient High-Resolution Neural Architectures with Lightweight Transformers
• Deep High-Resolution Representation Learning for Visual Recognition
• Deep High-Resolution Representation Learning for Visual Recognition
• Expediting Large-Scale Vision Transformer for Dense Prediction without Fine-tuning