2024/06/01 by Jordan K. Matelsky, Jordan Matelsky, Matelsky, Jordan K. +5 · 1 voice
Computer Science · #Artificial Intelligence (cs.AI) #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Neural Networks and Applications #Neural Networks and Reservoir Computing #cs.AI #cs.LG
paper · pdf · doi:10.48550/arxiv.2406.00509
openalex publication_date 2024/06/01 · arxiv published 2024/06/01 · arxiv updated 2024/06/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Understanding the process of learning in neural networks is crucial for improving their performance and interpreting their behavior. This can be approximately understood by asking how a model's output is influenced when we fine-tune on a new training sample. There are desiderata for such influences, such as decreasing influence with semantic distance, sparseness, noise invariance, transitive causality, and logical consistency. Here we use the empirical influence measured using fine-tuning to demonstrate how individual training samples affect outputs. We show that these desiderata are violated for both for simple convolutional networks and for a modern LLM. We also illustrate how prompting can partially rescue this failure. Our paper presents an efficient and practical way of quantifying how well neural networks learn from fine-tuning stimuli. Our results suggest that popular models cannot generalize or perform logic in the way they appear to.