SimLens for Early Exit in Large Language Models: Eliciting Accurate Latent Predictions with One More Token

2025/07/23 by Ming Ma, Zheng, Bowen, Bowen Zheng +5
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Performance (cs.PF) #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2507.17618

openalex publication_date 2025/07/23 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Intermediate-layer predictions in large language models (LLMs) are informative but hard to decode accurately, especially at early layers. Existing lens-style methods typically rely on direct linear readout, which is simple but often drifts away from the model's eventual prediction. We proposeSimLens, a simple training-free decoder for single-token decision tasks that keeps only the start token and a candidate answer token ([s] and [a]) and performs one lightweight continuation through the remaining upper layers. This surprisingly small modification recovers much more accurate latent predictions than direct linear decoding. We further introduce Linear SimLens, a lightweight linear approximation for entropy-based confidence estimation, and combine the two in SimExit, a hybrid early-exit mechanism. On ARC, BoolQ, and HeadQA with LLaMA-7B and Vicuna-7B, SimLens improves Iso-Compute accuracy in all six settings, with an average gain of +0.43 even when fair compute includes the extra two-token post-forward overhead. SimExit yields an average 1.15× speedup at the best-accuracy operating points and 1.40× when allowing up to a 1 percentage-point accuracy drop. Ablations show that [s] and [a] play distinct roles as global condition and semantic anchor, respectively.

Citations

• Why do LLMs attend to the first token?
• Position-Aware Depth Decay Decoding (D³): Boosting Large Language Model Inference Efficiency
• Performance Control in Early Exiting to Deploy Large Models at the Same Cost of Smaller Ones
• Optimizing Large Language Models through Quantization: A Comparative Analysis of PTQ and QAT Techniques
• Performance-Guided LLM Knowledge Distillation for Efficient Text Classification at Scale
• When Attention Sink Emerges in Language Models: An Empirical View
• Greedy Output Approximation: Towards Efficient Structured Pruning for LLMs Without Retraining
• Not All Layers of LLMs Are Necessary During Inference
• FlattenQuant: Breaking Through the Inference Compute-bound for Large Language Models with Per-tensor Quantization
• Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models
• In-Context Learning Creates Task Vectors
• Sorted LLaMA: Unlocking the Potential of Intermediate Layers of Large Language Models for Dynamic Inference
• Overthinking the Truth: Understanding how Language Models Process False Demonstrations
• Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learning
• Dissecting Recall of Factual Associations in Auto-Regressive Language Models
• Eliciting Latent Predictions from Transformers with the Tuned Lens
• Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
• HEAD-QA: A Healthcare Dataset for Complex Reasoning
• BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions
• Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
• Pointer Sentinel Mixture Models
• The Neural Basis of Decision Making
• Label Words as Local Task Vectors in In-Context Learning
• Accelerating Large Language Model Inference with Self-Supervised Early Exits

Related

• Deeper is Not Always Better: Mitigating the Alignment Tax via Confident Layer Decoding
• Confident Adaptive Language Modeling
• Fast Best-of-N Decoding via Speculative Rejection
• Generation Meets Verification: Accelerating Large Language Model Inference with Smart Parallel Auto-Correct Decoding
• Accelerating Large Language Model Inference with Self-Supervised Early Exits