Bridging the Semantic-Action Gap in Visual Token Pruning for Efficient VLA Inference
2025/11/20 by Liu, Ziyan, Chen, Yeqiu, Hongyi Cai +9
Computer Science · #Advanced Neural Network Applications #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Multimodal Machine Learning Applications
paper · pdf · doi:10.48550/arxiv.2511.16449
openalex publication_date 2025/11/20 · openalex created_date 2025/11/23 · openalex updated_date 2026/07/28
Abstract
Vision-Language-Action (VLA) models have shown great potential for embodied AI by integrating visual perception, language understanding, and action execution. In real-time deployment, these models must process continuous visual streams, incurring substantial computational overhead. Visual token pruning -- a mainstream technique for accelerating Vision-Language Models (VLMs) by retaining salient tokens while discarding redundant ones -- offers a natural candidate solution to this challenge. However, directly applying VLM-oriented pruning methods to VLA inference can cause severe degradation in manipulation performance. Our analysis attributes this degradation to a key mismatch: VLA inference exhibits distinct attention patterns between the vision-language prefill stage and the action-decode stage, so pruning based only on context-prefill semantic salience is biased toward semantic cues and may remove action-critical visual tokens. Motivated by this observation, we propose VLA-Pruner, an effective plug-and-play token pruning method grounded in the visual requirements of VLA inference, further exploiting the temporal continuity of robot manipulation. Specifically, VLA-Pruner estimates visual-token importance from both semantic prefilling and temporally smoothed action relevance, and then applies a Combine-then-Filter strategy to retain compact, non-redundant tokens under the compute budget. Experiments show that VLA-Pruner outperforms state-of-the-art approaches across multiple VLA architectures, achieving up to 1.99x speedup with comparable manipulation quality.
Citations
- Evo-1: Lightweight Vision-Language-Action Model with Preserved Semantic Alignment
- A Survey on Efficient Vision-Language-Action Models
- Don't Just Chase "Highlighted Tokens" in MLLMs: Revisiting Visual Holistic Context Retention
- SpecPrune-VLA: Accelerating Vision-Language-Action Models via Action-Aware Self-Speculative Pruning
- ThinkAct: Vision-Language-Action Reasoning via Reinforced Visual Latent Planning
- SP-VLA: A Joint Model Scheduling and Token Pruning Approach for VLA Model Acceleration
- EfficientVLA: Training-Free Acceleration and Compression for Vision-Language-Action Models
- OpenHelix: A Short Survey, Empirical Analysis, and Open-Source Dual-System VLA Model for Robotic Manipulation
- Efficient Token Compression for Vision Transformer with Spatial Information Preserved
- DivPrune: Diversity-based Visual Token Pruning for Large Multimodal Models
- Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success
- VLA-Cache: Efficient Vision-Language-Action Manipulation via Adaptive Token Caching
- Quantization-Aware Imitation-Learning for Resource-Efficient Robotic Control
- CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulation
- DeeR-VLA: Dynamic Inference of Multimodal Large Language Models for Efficient Robot Execution
- π0: A Vision-Language-Action Flow Model for General Robot Control
- PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reduction
- A Dual Process VLA: Efficient Robotic Manipulation Leveraging VLM
- SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference
- TinyVLA: Towards Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulation
- Fit and Prune: Fast and Training-free Visual Token Pruning for Multi-modal Large Language Models
- OpenVLA: An Open-Source Vision-Language-Action Model
- A Survey on Vision-Language-Action Models for Embodied AI
- Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference
- Evaluating Real-World Robot Manipulation Policies in Simulation
- LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models
- An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models
- Foundation Models in Robotics: Applications, Challenges, and the Future
- Vision-Language Foundation Models as Effective Robot Imitators
- Open X-Embodiment: Robotic Learning Datasets and RT-X Models
- Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs
- RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
- Llama 2: Open Foundation and Fine-Tuned Chat Models
- MMBench: Is Your Multi-modal Model an All-around Player?
- H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models
- LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning
- Visual Instruction Tuning
- Diffusion Policy: Visuomotor Policy Learning via Action Diffusion
- Diffusion policy: Visuomotor policy learning via action diffusion
- RT-1: Robotics Transformer for Real-World Control at Scale
- TGIF: A New Dataset and Benchmark on Animated GIF Description
- Flickr30k Entities: Collecting Region-to-Phrase Correspondences for\n Richer Image-to-Sentence Models
- Exponential smoothing: The state of the art—Part II
- Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy
- Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond
Related