OverFill: Two-Stage Models for Efficient Language Model Decoding

2025/08/11 by Woojeong Kim, Kim, Woojeong, Junxiong Wang +7
Computer Science · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2508.08446

openalex publication_date 2025/08/11 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Large language models (LLMs) excel across diverse tasks but face significant deployment challenges due to high inference costs. LLM inference comprises prefill (compute-bound) and decode (memory-bound) stages, with decode dominating latency particularly for long sequences. Current decoder-only models handle both stages uniformly, despite their distinct computational profiles. We propose OverFill, which decouples these stages to optimize accuracy-efficiency tradeoffs. OverFill begins with a full model for prefill, processing system and user inputs in parallel. It then switches to a dense pruned model, while generating tokens sequentially. Leveraging more compute during prefill, OverFill improves generation quality with minimal latency overhead. Our 3B-to-1B OverFill configuration outperforms 1B pruned models by 83.2%, while the 8B-to-3B configuration improves over 3B pruned models by 79.2% on average across standard benchmarks. OverFill matches the performance of same-sized models trained from scratch, while using significantly less training data. Our code is available at https://github.com/friendshipkim/overfill.

Citations

• Qwen2.5 Technical Report
• LLM Pruning and Distillation in Practice: The Minitron Approach
• MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decoding
• The Llama 3 Herd of Models
• LazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inference
• Are We Done with MMLU?
• You Only Cache Once: Decoder-Decoder Architectures for Language Models
• Kangaroo: Lossless Self-Speculative Decoding via Double Early Exiting
• Prepacking: A Simple Method for Fast Prefilling and Increased Throughput in Large Language Models
• LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression
• ShortGPT: Layers in Large Language Models are More Redundant Than You Expect
• Think Big, Generate Quick: LLM-to-SLM for Fast Autoregressive Decoding
• LaCo: Large Language Model Pruning via Layer Collapse
• Tandem Transformers for Inference Efficient LLMs
• Everybody Prune Now: Structured Pruning of LLMs with only Forward Passes
• Hydra: Sequentially-Dependent Draft Heads for Medusa Decoding
• Shortened LLaMA: Depth Pruning for Large Language Models with Comparison of Retraining Methods
• KIVI : Plug-and-play 2bit KV Cache Quantization with Streaming Asymmetric Quantization
• GliDe with a CaPE: A Low-Hassle Method to Accelerate Speculative Decoding
• EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty
• SliceGPT: Compress Large Language Models by Deleting Rows and Columns
• DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving
• CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution
• Splitwise: Efficient generative LLM inference using phase splitting
• Instruction-Following Evaluation for Large Language Models
• DistillSpec: Improving Speculative Decoding via Knowledge Distillation
• Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning
• Efficient Streaming Language Models with Attention Sinks
• H₂O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models
• AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
• LLM-Pruner: On the Structural Pruning of Large Language Models
• FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU
• Fast Inference from Transformers via Speculative Decoding
• SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
• GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
• Prompt Compression and Contrastive Conditioning for Controllability and Toxicity Reduction in Language Models
• Confident Adaptive Language Modeling
• Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation
• Training Verifiers to Solve Math Word Problems
• Instantaneous Grammatical Error Correction with Shallow Aggressive Decoding
• Measuring Mathematical Problem Solving With the MATH Dataset
• Measuring Massive Multitask Language Understanding
• Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge

Related

• Prepacking: A Simple Method for Fast Prefilling and Increased Throughput in Large Language Models
• SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills
• When Compression Meets Model Compression: Memory-Efficient Double Compression for Large Language Models
• Large Language Models Are Overparameterized Text Encoders
• InfiniteHiP: Extending Language Model Context Up to 3 Million Tokens on a Single GPU