Task Matrices: Linear Maps for Cross-Model Finetuning Transfer

2025/12/16 by Brien, Darrin O', Gajulapalli, Dhikshith, Xia, Eric
Computer Science · #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Multimodal Machine Learning Applications #Topic Modeling

paper · doi:10.48550/arxiv.2512.14880

openalex publication_date 2025/12/16 · openalex created_date 2025/12/19 · openalex updated_date 2026/07/28

Abstract

Results in interpretability suggest that large vision and language models learn implicit linear encodings when models are biased by in-context prompting. However, the existence of similar linear representations in more general adaptation regimes has not yet been demonstrated. In this work, we develop the concept of a task matrix, a linear transformation from a base to finetuned embedding state. We demonstrate that for vision and text models and ten different datasets, a base model augmented with a task matrix achieves results surpassing linear probes, sometimes approaching finetuned levels. Our results validate the existence of cross-layer linear encodings between pretrained and finetuned architectures. Moreover, we show that a data-based approximation for such encodings is both efficient and generalizable to multiple domains. We make our implementation publicly available.

Citations

• Linear Relational Decoding of Morphology in Language Models
• Identifying Linear Relational Concepts in Large Language Models
• The Linear Representation Hypothesis and the Geometry of Large Language Models
• Linearity of Relation Decoding in Transformer Language Models
• Dissecting Recall of Factual Associations in Auto-Regressive Language Models
• Double Descent Demystified: Identifying, Interpreting & Ablating the Sources of a Deep Learning Puzzle
• AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning
• High-dimensional analysis of double descent for linear regression with random projections
• Editing Models with Task Arithmetic
• Mass-Editing Memory in a Transformer
• Locating and Editing Factual Associations in GPT
• BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models
• LoRA: Low-Rank Adaptation of Large Language Models
• Benchmarking Representation Learning for Natural World Image Collections
• Learning Transferable Visual Models From Natural Language Supervision
• Transformer Feed-Forward Layers Are Key-Value Memories
• Training data-efficient image transformers & distillation through attention
• Better Fine-Tuning by Reducing Representational Collapse
• Efficient Intent Detection with Dual Sentence Encoders
• MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
• BLiMP: The Benchmark of Linguistic Minimal Pairs for English
• The generalization error of random features regression: Precise asymptotics and double descent curve
• The Generalization Error of Random Features Regression: Precise Asymptotics and the Double Descent Curve
• Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference
• EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification
• Attention Is All You Need
• Understanding intermediate layers using linear classifier probes
• A large annotated corpus for learning natural language inference
• Efficient Estimation of Word Representations in Vector Space
• Ridge Regression: Biased Estimation for Nonorthogonal Problems
• FusionBench: A Unified Library and Comprehensive Benchmark for Deep Model Fusion
• Prefix-Tuning: Optimizing Continuous Prompts for Generation
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Related

• Vision-Language Models Create Cross-Modal Task Representations
• Task Vectors, Learned Not Extracted: Performance Gains and Mechanistic Insight
• Rethinking Fine-Tuning: Unlocking Hidden Capabilities in Vision-Language Models
• Enhancing Target-unspecific Tasks through a Features Matrix
• Layer by Layer: Uncovering Where Multi-Task Learning Happens in Instruction-Tuned Large Language Models