Parallel Token Prediction for Language Models

2025/12/24 by Felix Draxler, Justus C. Will, Draxler, Felix +9 · 1 citation
Computer Science · #Autoregressive model #Bottleneck #Computation and Language (cs.CL) #Decoding methods #FOS: Computer and information sciences #Language model #Machine Learning (cs.LG) #Natural Language Processing Techniques #Security token #Sequence (biology) #Speech Recognition and Synthesis #Topic Modeling #Transformer

paper · doi:10.48550/arxiv.2512.21323

published in Open MIND

openalex publication_date 2025/12/24 · openalex created_date 2025/12/26 · openalex updated_date 2026/07/28

Abstract

Autoregressive decoding in language models is inherently slow, generating only one token per forward pass. We propose Parallel Token Prediction (PTP), a general-purpose framework for predicting multiple tokens in a single model call. PTP moves the source of randomness from post-hoc sampling to random input variables, making future tokens deterministic functions of those inputs and thus jointly predictable in a single forward pass. We prove that a single PTP call can represent arbitrary dependencies between tokens. PTP is trained by distilling an existing model or through inverse autoregressive training without a teacher. Experimentally, PTP achieves a 2.4x speedup on a diverse-task speculative decoding benchmark. We provide code and checkpoints at https://github.com/mandt-lab/ptp.

Citations

• Your LLM Knows the Future: Uncovering Its Multi-Token Prediction Potential
• Theoretical Benefit and Limitation of Diffusion Language Model
• Speeding up Speculative Decoding via Sequential Approximate Verification
• SAM Decoding: Speculative Decoding via Suffix Automaton
• Discrete Copula Diffusion
• Turning Trash into Treasure: Accelerating Inference of Large Language Models with Token Recycling
• EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees
• A Comprehensive Survey of Accelerated Generation Techniques in Large Language Models
• Better & Faster Large Language Models via Multi-token Prediction
• Hydra: Sequentially-Dependent Draft Heads for Medusa Decoding
• Break the Sequential Dependency of LLM Inference Using Lookahead Decoding
• EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty
• Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads
• Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding
• TinyLlama: An Open-Source Small Language Model
• ShareGPT4V: Improving Large Multi-Modal Models with Better Captions
• Free-form Flows: Make Any Architecture a Normalizing Flow
• SpecTr: Fast Speculative Decoding via Optimal Transport
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• EasyTPP: Towards Open Benchmarking Temporal Point Processes
• StarCoder: may the source be with you!
• Accelerating Large Language Model Decoding with Speculative Sampling
• Fast Inference from Transformers via Speculative Decoding
• Structured Denoising Diffusion Models in Discrete State-Spaces
• Argmax Flows and Multinomial Diffusion: Learning Categorical Distributions
• Language Models are Few-Shot Learners
• ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• The Curious Case of Neural Text Degeneration
• Parallel WaveNet: Fast High-Fidelity Speech Synthesis
• Non-Autoregressive Neural Machine Translation
• On Calibration of Modern Neural Networks
• Attention Is All You Need
• Improving Variational Inference with Inverse Autoregressive Flow
• Distilling the Knowledge in a Neural Network

Cited by

• Multi-Token Prediction via Self-Distillation

Related

• Fast Inference from Transformers via Speculative Decoding
• Exploring and Improving Drafts in Blockwise Parallel Decoding
• Faster Language Models with Better Multi-Token Prediction Using Tensor Decomposition
• Reviving Any-Subset Autoregressive Models with Principled Parallel Sampling and Speculative Decoding
• PaSS: Parallel Speculative Sampling