MAGIC: Near-Optimal Data Attribution for Deep Learning

2025/04/23 by Andrew Ilyas, Logan Engstrom, Ilyas, Andrew +1 · 4 citations
Computer Science · #Neural Networks and Applications

paper · pdf · doi:10.48550/arxiv.2504.16430

Abstract

The goal of predictive data attribution is to estimate how adding or removing a given set of training datapoints will affect model predictions. In convex settings, this goal is straightforward (i.e., via the infinitesimal jackknife). In large-scale (non-convex) settings, however, existing methods are far less successful -- current methods' estimates often only weakly correlate with ground truth. In this work, we present a new data attribution method (MAGIC) that combines classical methods and recent advances in metadifferentiation to (nearly) optimally estimate the effect of adding or removing training data on model predictions.

Cited by

• Position: Don't Just "Fix it in Post": A Science of AI Must Study Training Dynamics
• Small edits, large models: How Wikipedia advocacy shapes LLM values
• Better Hessians Matter: Studying the Impact of Curvature Approximations in Influence Functions
• Train Once, Answer All: Many Pretraining Experiments for the Cost of One

Related

• TRAK: Attributing Model Behavior at Scale
• Efficient Ensembles Improve Training Data Attribution
• Natural Geometry of Robust Data Attribution: From Convex Models to Deep Networks
• Improving performance of deep learning models with axiomatic attribution priors and expected gradients
• Daunce: Data Attribution through Uncertainty Estimation