Compressing Large Language Models with PCA Without Performance Loss

2025/08/06 by Bengtsson, Magnus
#Artificial Intelligence (cs.AI) #Computational Engineering #FOS: Computer and information sciences #Finance #and Science (cs.CE)

paper · doi:10.48550/arxiv.2508.04307

Abstract

We demonstrate that Principal Component Analysis (PCA), when applied in a structured manner, either to polar-transformed images or segment-wise to token sequences, enables extreme compression of neural models without sacrificing performance. Across three case studies, we show that a one-layer classifier trained on PCA-compressed polar MNIST achieves over 98 percent accuracy using only 840 parameters. A two-layer transformer trained on 70-dimensional PCA-reduced MiniLM embeddings reaches 76.62 percent accuracy on the 20 Newsgroups dataset with just 81000 parameters. A decoder-only transformer generates coherent token sequences from 70-dimensional PCA embeddings while preserving over 97 percent cosine similarity with full MiniLM representations, using less than 17 percent of the parameter count of GPT-2. These results highlight PCA-based input compression as a general and effective strategy for aligning model capacity with information content, enabling lightweight architectures across multiple modalities.

Citations

• Knowledge Base Index Compression via Dimensionality and Precision Reduction
• On the Downstream Performance of Compressed Word Embeddings
• Single Training Dimension Selection for Word Embedding with PCA
• FRAGE: Frequency-Agnostic Word Representation
• Efficient Estimation of Word Representations in Vector Space
• Gradient-based learning applied to document recognition
• Analysis of a complex of statistical variables into principal components.

Related

• MatryoshkaKV: Adaptive KV Compression via Trainable Orthogonal Projection
• LayerCollapse: Adaptive compression of neural networks
• LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximation
• Neural Weight Compression for Language Models
• Lossless Compression of Neural Network Components: Weights, Checkpoints, and K/V Caches in Low-Precision Formats