LatentLLM: Attention-Aware Joint Tensor Compression

2025/05/23 by Koike-Akino, Toshiaki, Chen, Xiangyu, Liu, Jing +4 · 3 citations
#Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2505.18413

Abstract

Modern foundation models such as large language models (LLMs) and large multi-modal models (LMMs) require a massive amount of computational and memory resources. We propose a new framework to convert such LLMs/LMMs into a reduced-dimension latent structure. Our method extends a local activation-aware tensor decomposition to a global attention-aware joint tensor de-composition. Our framework can significantly improve the model accuracy over the existing model compression methods when reducing the latent dimension to realize computationally/memory-efficient LLMs/LLMs. We show the benefit on several benchmark including multi-modal reasoning tasks.

Cited by

• SkipCat: Rank-Maximized Low-Rank Compression of Large Language Models via Shared Projection and Block Skipping
• UniQL: Unified Quantization and Low-rank Compression for Adaptive Edge LLMs
• Globally optimized SVD compression of LLMs via Fermi-function-based rank selection and gauge fixing

Related

• TensorLLM: Tensorising Multi-Head Attention for Enhanced Reasoning and Compression in LLMs
• Communication Compression for Tensor Parallel LLM Inference
• DoTA: Weight-Decomposed Tensor Adaptation for Large Language Models
• Geometry is All You Need: A Unified Taxonomy of Matrix and Tensor Factorization for Compression of Generative Language Models
• ESPACE: Dimensionality Reduction of Activations for Model Compression