Parallel Token Prediction for Language Models

2025/12/24 by Draxler, Felix, Will, Justus, Sofian, Farrin Marouf +3
#Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2512.21323

Abstract

We propose Parallel Token Prediction (PTP), a universal framework for parallel sequence generation in language models. PTP jointly predicts multiple dependent tokens in a single transformer call by incorporating the sampling procedure into the model. This reduces the latency bottleneck of autoregressive decoding, and avoids the restrictive independence assumptions common in existing multi-token prediction methods. We prove that PTP can represent arbitrary autoregressive sequence distributions. PTP is trained either by distilling an existing model or through inverse autoregressive training without a teacher. Experimentally, we achieve state-of-the-art speculative decoding performance on Vicuna-7B by accepting over four tokens per step on Spec-Bench. The universality of our framework indicates that parallel generation of long sequences is feasible without loss of modeling power.

Citations

• Your LLM Knows the Future: Uncovering Its Multi-Token Prediction Potential
• Theoretical Benefit and Limitation of Diffusion Language Model
• Speeding up Speculative Decoding via Sequential Approximate Verification
• SAM Decoding: Speculative Decoding via Suffix Automaton
• Discrete Copula Diffusion
• Turning Trash into Treasure: Accelerating Inference of Large Language Models with Token Recycling
• EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees
• A Comprehensive Survey of Accelerated Generation Techniques in Large Language Models
• Better & Faster Large Language Models via Multi-token Prediction
• Hydra: Sequentially-Dependent Draft Heads for Medusa Decoding
• Break the Sequential Dependency of LLM Inference Using Lookahead Decoding
• EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty
• Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads
• Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding
• TinyLlama: An Open-Source Small Language Model
• ShareGPT4V: Improving Large Multi-Modal Models with Better Captions
• Free-form Flows: Make Any Architecture a Normalizing Flow
• SpecTr: Fast Speculative Decoding via Optimal Transport
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• EasyTPP: Towards Open Benchmarking Temporal Point Processes
• StarCoder: may the source be with you!
• Accelerating Large Language Model Decoding with Speculative Sampling
• Fast Inference from Transformers via Speculative Decoding
• Structured Denoising Diffusion Models in Discrete State-Spaces
• Argmax Flows and Multinomial Diffusion: Learning Categorical Distributions
• Language Models are Few-Shot Learners
• ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• The Curious Case of Neural Text Degeneration
• Parallel WaveNet: Fast High-Fidelity Speech Synthesis
• Non-Autoregressive Neural Machine Translation
• On Calibration of Modern Neural Networks
• Attention Is All You Need
• Improving Variational Inference with Inverse Autoregressive Flow
• Distilling the Knowledge in a Neural Network

Related

• Fast Inference from Transformers via Speculative Decoding
• Exploring and Improving Drafts in Blockwise Parallel Decoding
• Faster Language Models with Better Multi-Token Prediction Using Tensor Decomposition
• Reviving Any-Subset Autoregressive Models with Principled Parallel Sampling and Speculative Decoding
• PaSS: Parallel Speculative Sampling