CascadedViT: Cascaded Chunk-FeedForward and Cascaded Group Attention Vision Transformer

2025/11/18 by Sivakumar, Srivathsan, Qureshi, Faisal Z.
#Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2511.14111

Abstract

Vision Transformers (ViTs) have demonstrated remarkable performance across a range of computer vision tasks; however, their high computational, memory, and energy demands hinder deployment on resource-constrained platforms. In this paper, we propose Cascaded-ViT (CViT), a lightweight and compute-efficient vision transformer architecture featuring a novel feedforward network design called Cascaded-Chunk Feed Forward Network (CCFFN). By splitting input features, CCFFN improves parameter and FLOP efficiency without sacrificing accuracy. Experiments on ImageNet-1K show that our CViT-XL model achieves 75.5% Top-1 accuracy while reducing FLOPs by 15% and energy consumption by 3.3% compared to EfficientViT-M5. Across various model sizes, the CViT family consistently exhibits the lowest energy consumption, making it suitable for deployment on battery-constrained devices such as mobile phones and drones. Furthermore, when evaluated using a new metric called Accuracy-Per-FLOP (APF), which quantifies compute efficiency relative to accuracy, CViT models consistently achieve top-ranking efficiency. Particularly, CViT-L is 2.2% more accurate than EfficientViT-M2 while having comparable APF scores.

Citations

• Learning Occlusion-Robust Vision Transformers for Real-Time UAV Tracking
• Vision Transformer Accelerator ASIC for Real-Time Low-Power Sleep Staging
• MicroViT: A Vision Transformer with Low Complexity Self Attention for Edge Device
• CMoE: Converting Mixture-of-Experts from Dense to Accelerate LLM Inference
• Double-Exponential Increases in Inference Energy: The Cost of the Race\n for Accuracy
• Does Self-Attention Need Separate Weights in Transformers?
• Vision Transformers for End-to-End Vision-Based Quadrotor Obstacle Avoidance
• EfficientASR: Speech Recognition Network Compression via Attention Redundancy and Chunk-Level FFN Optimization
• A Survey on Transformer Compression
• SHViT: Single-Head Vision Transformer with Memory Efficient Macro Design
• Can pruning make Large Language Models more efficient?
• ResidualTransformer: Residual Low-Rank Learning with Weight-Sharing for Transformer Layers
• Compressing Vision Transformers for Low-Resource Visual Learning
• One Wide Feedforward is All You Need
• EfficientViT: Memory Efficient Vision Transformer with Cascaded Group Attention
• Reversible Column Networks
• MobileViTv3: Mobile-Friendly Vision Transformer with Simple and Effective Fusion of Local, Global and Input Features
• TinyViT: Fast Pretraining Distillation for Small Vision Transformers
• LightViT: Towards Light-Weight Convolution-Free Vision Transformers
• EdgeNeXt: Efficiently Amalgamated CNN-Transformer Architecture for Mobile Vision Applications
• Separable Self-attention for Mobile Vision Transformers
• EfficientFormer: Vision Transformers at MobileNet Speed
• Knowledge Distillation from A Stronger Teacher
• EdgeViTs: Competing Light-weight CNNs on Mobile Devices with Vision Transformers
• MiniViT: Compressing Vision Transformers with Weight Multiplexing
• SPViT: Enabling Faster Vision Transformers via Soft Token Pruning
• Global Vision Transformer Pruning with Hessian-Aware Saliency
• Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers
• Patch Slimming for Efficient Vision Transformers
• DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification
• Vision Transformer Pruning
• LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inference
• CvT: Introducing Convolutions to Vision Transformers
• Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions
• Dancing along Battery: Enabling Transformer with Run-time Reconfigurability on Mobile Devices
• Transformer Feed-Forward Layers Are Key-Value Memories
• Training data-efficient image transformers & distillation through attention
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Language Models are Few-Shot Learners
• Scaling Laws for Neural Language Models
• Deeper Insights into Weight Sharing in Neural Architecture Search
• Green AI
• EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
• Searching for MobileNetV3
• mixup: Beyond Empirical Risk Minimization
• AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baseline
• Squeeze-and-Excitation Networks
• Random Erasing Data Augmentation
• Mask R-CNN
• Soft Weight-Sharing for Neural Network Compression
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Microsoft COCO: Common Objects in Context
• Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

Related

• Vision Transformers for Mobile Applications: A Short Survey
• EfficientViT: Memory Efficient Vision Transformer with Cascaded Group Attention
• MicroViT: A Vision Transformer with Low Complexity Self Attention for Edge Device
• Rethinking Vision Transformers for MobileNet Size and Speed
• CvT: Introducing Convolutions to Vision Transformers