You Had One Job: Per-Task Quantization Using LLMs' Hidden Representations

2025/11/09 by LeVi, Amit, Lapid, Raz, Baskin, Chaim +4
Computer Science · #Advanced Neural Network Applications #Computation and Language (cs.CL) #Data compression #Domain Adaptation and Few-Shot Learning #Entropy (arrow of time) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Noise measurement #Pattern recognition (psychology) #Quantization (signal processing) #Sensitivity (control systems) #Set (abstract data type)

paper · doi:10.48550/arxiv.2511.06516

published in Open MIND

openalex publication_date 2025/11/09 · openalex created_date 2025/11/12 · openalex updated_date 2026/07/28

Abstract

Many LLM applications require only narrow capabilities, yet standard post-training quantization (PTQ) methods allocate precision without considering the target task. This can waste bits on layers that are less relevant to the task signal while over-compressing layers that are critical for downstream behavior. We propose Task-Aware Quantization (TAQ), a training-free, weight-only mixed-precision PTQ framework that uses a small set of unlabeled task calibration prompts to allocate higher precision to task-relevant transformer layers under a fixed bit budget. TAQ estimates layer importance from hidden representations and output sensitivity, and we instantiate it with three scoring rules: TAQ-IS, based on activation information and stability; TAQ-KL, based on output-distribution sensitivity under a quantization-noise proxy; and TAQ-O, a label-informed oracle diagnostic for analyzing layer sensitivity. Across several benchmarks, TAQ outperforms task-agnostic baselines such in most settings, with especially strong gains in the accuracy--memory ratio. We further validate that these gains translate to real deployment behavior through hardware throughput and latency measurements, and analyze calibration robustness and residual-stream error propagation. Overall, TAQ turns mixed-precision PTQ from a model-centric compression step into a task-conditioned precision-allocation problem. A reference implementation is available at \hrefhttps://anonymous.4open.science/r/TAQ-9217/README.md\includegraphics[height=1em]imgs/github-mark.png.

Citations

• Silenced Biases: The Dark Side LLMs Learned to Refuse
• CoopQ: Cooperative Game Inspired Layerwise Mixed Precision Quantization for LLMs
• Towards Superior Quantization Accuracy: A Layer-sensitive Approach
• Jailbreak Attack Initializations as Extractors of Compliance Directions
• Layer by Layer: Uncovering Hidden Representations in Language Models
• Open Problems in Mechanistic Interpretability
• Qwen2.5 Technical Report
• MixLLM: LLM Quantization with Global Mixed-precision between Output-features and Highly-efficient System Design
• Phi-4 Technical Report
• CodeMMLU: A Multi-Task Benchmark for Assessing Code Understanding & Reasoning Capabilities of CodeLLMs
• From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networks
• Investigating Layer Importance in Large Language Models
• Qwen2.5-Coder Technical Report
• Safety Layers in Aligned Large Language Models: The Key to LLM Security
• The Llama 3 Herd of Models
• Gemma 2: Improving Open Language Models at a Practical Size
• Qwen2 Technical Report
• External Invariants: A Cryptographic Trust Architecture for Institutional AI Inference
• MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark
• SpinQuant: LLM quantization with learned rotations
• SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Models
• Post Training Quantization of Large Language Models with Microscaling Formats
• How to use and interpret activation patching
• In-Context Learning Creates Task Vectors
• Representation Engineering: A Top-Down Approach to AI Transparency
• Sparse Autoencoders Find Highly Interpretable Features in Language Models
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models
• Code Llama: Open Foundation Models for Code
• Steering Language Models With Activation Engineering
• A Survey on Model Compression for Large Language Models
• SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression
• AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
• Improving Convergence and Generalization Using Parameter Symmetries
• RPTQ: Reorder-based Post-training Quantization for Large Language Models
• Editing Models with Task Arithmetic
• Discovering Latent Knowledge in Language Models Without Supervision
• SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
• Efficiently Scaling Transformer Inference
• Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small
• GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
• Polysemanticity and Capacity in Neural Networks
• Relative representations enable zero-shot latent space communication
• LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale
• ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
• Training Compute-Optimal Large Language Models
• Training language models to follow instructions with human feedback
• Locating and Editing Factual Associations in GPT
• Training Verifiers to Solve Math Word Problems
• Generalized Shape Metrics on Neural Representations
• Program Synthesis with Large Language Models
• Evaluating Large Language Models Trained on Code
• A White Paper on Neural Network Quantization
• A Survey of Quantization Methods for Efficient Neural Network Inference
• Measuring Mathematical Problem Solving With the MATH Dataset
• Transformer Feed-Forward Layers Are Key-Value Memories
• Up or Down? Adaptive Rounding for Post-Training Quantization
• Scaling Laws for Neural Language Models
• DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
• Quantizing deep convolutional networks for efficient inference: A whitepaper
• Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference
• Attention Is All You Need
• TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for\n Reading Comprehension
• SQuAD: 100,000+ Questions for Machine Comprehension of Text
• Distilling the Knowledge in a Neural Network
• ZeroQuant-V2: Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensation

Cited by

• Silenced Biases: The Dark Side LLMs Learned to Refuse

Related

• Task-Circuit Quantization: Leveraging Knowledge Localization and Interpretability for Compression
• Exploring the Trade-Offs: Quantization Methods, Task Difficulty, and Model Size in Large Language Models From Edge to Giant
• QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Models
• Enhancing Computation Efficiency in Large Language Models through Weight and Activation Quantization
• SASQ: Static Activation Scaling for Quantization-Aware Training in Large Language Models