VVS: Accelerating Speculative Decoding for Visual Autoregressive Generation via Partial Verification Skipping

2025/11/17 by Dong, Haotian, Li, Ye, Lu, Rongwei +3
Computer Science · #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #Digital Media Forensic Detection #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #Image Enhancement Techniques

paper · doi:10.48550/arxiv.2511.13587

openalex publication_date 2025/11/17 · openalex created_date 2025/11/19 · openalex updated_date 2026/07/28

Abstract

Visual autoregressive (AR) generation models have demonstrated strong potential for image generation, yet their next-token-prediction paradigm introduces considerable inference latency. Although speculative decoding (SD) has been proven effective for accelerating visual AR models, its "draft one step, then verify one step" paradigm prevents a direct reduction in the number of forward passes, limiting its acceleration potential. Motivated by the interchangeability of visual tokens, we explore verification skipping in the SD process for the first time to explicitly cut the number of target model forward passes, thereby reducing inference latency. By analyzing the characteristics of the drafting stage, we observe that verification redundancy and stale feature reusability are key factors to maintain generation quality while improving speed for verification-free steps. Inspired by these two observations, we propose a novel SD framework VVS to accelerate visual AR model via partial verification skipping, which integrates three complementary modules: (1) a verification-free token selector with dynamic truncation, (2) token-level feature caching and reuse, and (3) fine-grained skipped step scheduling. Consequently, VVS reduces the number of target model forward passes by 2.8× relative to vanilla AR decoding while maintaining competitive generation quality, offering a superior speed-quality trade-off over conventional SD frameworks and revealing strong potential to reshape the SD paradigm. Our code is available at https://github.com/HyattDD/VVS.

Citations

• Grouped Speculative Decoding for Autoregressive Image Generation
• Lumina-mGPT 2.0: Stand-Alone AutoRegressive Image Modeling
• SP-VLA: A Joint Model Scheduling and Token Pruning Approach for VLA Model Acceleration
• Beyond A Single AI Cluster: A Survey of Decentralized LLM Training
• EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test
• Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignment
• ZipAR: Parallel Auto-regressive Image Generation through Spatial Locality
• Collaborative Decoding Makes Visual Auto-Regressive Modeling Efficient
• Staleness-Centric Optimizations for Parallel Diffusion MoE Inference
• LANTERN: Accelerating Visual Autoregressive Models with Relaxed Speculative Decoding
• Accelerating Auto-regressive Text-to-Image Generation with Training-free Speculative Jacobi Decoding
• Lumina-mGPT: Illuminate Flexible Photorealistic Text-to-Image Generation with Multimodal Generative Pretraining
• ANOLE: An Open, Autoregressive, Native Large Multimodal Models for Interleaved Image-Text Generation
• PRANCE: Joint Token-Optimization and Structural Channel-Pruning for Adaptive ViT Inference
• EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees
• Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation
• Break the Sequential Dependency of LLM Inference Using Lookahead Decoding
• EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty
• Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads
• Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding
• SpecTr: Fast Speculative Decoding via Optimal Transport
• SkipDecode: Autoregressive Skip Decoding with Batching and Caching for Efficient LLM Inference
• Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis
• Accelerating Large Language Model Decoding with Speculative Sampling
• Fast Inference from Transformers via Speculative Decoding
• Diffusion Models: A Comprehensive Survey of Methods and Applications
• Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation
• Mixed-Precision Neural Network Quantization via Learned Layer-wise Importance
• Vector-quantized Image Modeling with Improved VQGAN
• Denoising Diffusion Probabilistic Models
• Language Models are Few-Shot Learners
• Improved Precision and Recall Metric for Assessing Generative Models
• Blockwise Parallel Decoding for Deep Autoregressive Models
• Microsoft COCO: Common Objects in Context

Related

• LANTERN: Accelerating Visual Autoregressive Models with Relaxed Speculative Decoding
• Grouped Speculative Decoding for Autoregressive Image Generation
• Continuous Speculative Decoding for Autoregressive Image Generation
• Spec-LLaVA: Accelerating Vision-Language Models with Dynamic Tree-Based Speculative Decoding
• FastVLM: Self-Speculative Decoding for Fast Vision-Language Model Inference