Saten: Sparse Augmented Tensor Networks for Post-Training Compression of Large Language Models

2025/05/20 by Ryan Solgi, Kai Zhen, Solgi, Ryan +11 · 1 citation
Computer Science · Mathematics · #Advanced Neural Network Applications #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Tensor decomposition and applications #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2505.14871

openalex publication_date 2025/05/20 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The efficient implementation of large language models (LLMs) is crucial for deployment on resource-constrained devices. Low-rank tensor compression techniques, such as tensor-train (TT) networks, have been widely studied for over-parameterized neural networks. However, their applications to compress pre-trained large language models (LLMs) for downstream tasks (post-training) remains challenging due to the high-rank nature of pre-trained LLMs and the lack of access to pretraining data. In this study, we investigate low-rank tensorized LLMs during fine-tuning and propose sparse augmented tensor networks (Saten) to enhance their performance. The proposed Saten framework enables full model compression. Experimental results demonstrate that Saten enhances both accuracy and compression efficiency in tensorized language models, achieving state-of-the-art performance.

Cited by

• INTERLACE: Interleaved Layer Pruning and Efficient Adaptation in Large Vision-Language Models

Related

• TensorGPT: Efficient Compression of Large Language Models based on Tensor-Train Decomposition
• Tensor Train Low-rank Approximation (TT-LoRA): Democratizing AI with Accelerated LLMs
• Tensorized Embedding Layers for Efficient Model Compression
• CompactifAI: Extreme Compression of Large Language Models using Quantum-Inspired Tensor Networks
• TQCompressor: improving tensor decomposition methods in neural networks via permutations