2023/03/13 by Feng Li, Li, Feng, Ailing Zeng +11 · 8 citations
Computer Science · #Advanced Neural Network Applications #Artificial intelligence #Cartography #Computer science #Data mining #Domain Adaptation and Few-Shot Learning #Encoder #Feature (linguistics) #Scale (ratio) #Video Surveillance and Tracking Methods
paper · pdf · doi:10.48550/arxiv.2303.07335
published in arXiv (Cornell University) (Cornell University)
openalex publication_date 2023/03/13 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05
Recent DEtection TRansformer-based (DETR) models have obtained remarkable performance. Its success cannot be achieved without the re-introduction of multi-scale feature fusion in the encoder. However, the excessively increased tokens in multi-scale features, especially for about 75% of low-level features, are quite computationally inefficient, which hinders real applications of DETR models. In this paper, we present Lite DETR, a simple yet efficient end-to-end object detection framework that can effectively reduce the GFLOPs of the detection head by 60% while keeping 99% of the original performance. Specifically, we design an efficient encoder block to update high-level features (corresponding to small-resolution feature maps) and low-level features (corresponding to large-resolution feature maps) in an interleaved way. In addition, to better fuse cross-scale features, we develop a key-aware deformable attention to predict more reliable attention weights. Comprehensive experiments validate the effectiveness and efficiency of the proposed Lite DETR, and the efficient encoder strategy can generalize well across existing DETR-based models. The code will be available in \urlhttps://github.com/IDEA-Research/Lite-DETR.