P²HCT: Plug-and-Play Hierarchical C2F Transformer for Multi-Scale Feature Fusion

2025/05/19 by Junyi Hu, Hu, Junyi, Tian Bai +7
Computer Science · #Advanced Image Processing Techniques #Advanced Neural Network Applications #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences

paper · pdf · doi:10.48550/arxiv.2505.12772

openalex publication_date 2025/05/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/04

Abstract

Feature fusion plays a pivotal role in achieving high performance in vision models, yet existing attention-based fusion techniques often suffer from substantial computational overhead and implementation complexity, particularly in resource-constrained settings. To address these limitations, we introduce the Plug-and-Play Hierarchical C2F Transformer (P²HCT), a lightweight module that combines coarse-to-fine token selection with shared attention parameters to preserve spatial details while reducing inference cost. P²HCT is trainable using coarse attention alone and can be seamlessly activated at inference to enhance accuracy without retraining. Integrated into real-time detectors such as YOLOv11-N/S/M, P²HCT achieves mAP gains of 0.9%, 0.5%, and 0.4% on MS COCO with minimal latency increase. Similarly, embedding P²HCT into ResNet-18/50/101 backbones improves ImageNet top-1 accuracy by 6.5%, 1.7%, and 1.0%, respectively. These results underscore P²HCT's effectiveness as a hardware-friendly and general-purpose enhancement for both detection and classification tasks.

Citations

• Cross-Layer Feature Pyramid Transformer for Small Object Detection in Aerial Images
• YOLOv9: Learning What You Want to Learn Using Programmable Gradient Information
• Adaptive Sparse ViT: Towards Learnable Adaptive Token Pruning by Fully Exploiting Self-Attention
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
• GiraffeDet: A Heavy-Neck Paradigm for Object Detection
• DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification
• A²-FPN: Attention Aggregation based Feature Pyramid Network for Instance Segmentation
• Conditional Positional Encodings for Vision Transformers
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Deformable DETR: Deformable Transformers for End-to-End Object Detection
• Rethinking Attention with Performers
• EfficientDet: Scalable and Efficient Object Detection
• EfficientDet: Scalable and Efficient Object Detection
• Attention Is All You Need
• Feature Pyramid Networks for Object Detection
• Microsoft COCO: Common Objects in Context

Related

• Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions
• Vision Transformer with Sparse Scan Prior
• Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions
• Sparse-Tuning: Adapting Vision Transformers with Efficient Fine-tuning and Inference
• SaiT: Sparse Vision Transformers through Adaptive Token Pruning