Transformers Handle Endogeneity in In-Context Linear Regression

2024/10/02 by Krishnakumar Balasubramanian, Liang, Haodong, Balasubramanian, Krishnakumar +2 · 5 citations
Computer Science · #Face and Expression Recognition #Neural Networks and Applications

paper · pdf · doi:10.48550/arxiv.2410.01265

Abstract

We explore the capability of transformers to address endogeneity in in-context linear regression. Our main finding is that transformers inherently possess a mechanism to handle endogeneity effectively using instrumental variables (IV). First, we demonstrate that the transformer architecture can emulate a gradient-based bi-level optimization procedure that converges to the widely used two-stage least squares (\textsf2SLS) solution at an exponential rate. Next, we propose an in-context pretraining scheme and provide theoretical guarantees showing that the global minimizer of the pre-training loss achieves a small excess loss. Our extensive experiments validate these theoretical findings, showing that the trained transformer provides more robust and reliable in-context predictions and coefficient estimates than the \textsf2SLS method, in the presence of endogeneity.

Cited by

• Differentially Private Two-Stage Gradient Descent for Instrumental Variable Regression
• Evidence-Type Competition: When Can Interventional Data Teach Language Models Causal Direction?
• CausalPFN: Amortized Causal Effect Estimation via In-Context Learning
• Interpretability and Generalization Bounds for Learning Spatial Physics
• How Private is Your Attention? Bridging Privacy with In-Context Learning

Related

• How Well Can Transformers Emulate In-context Newton's Method?
• Transformers Learn to Achieve Second-Order Convergence Rates for In-Context Linear Regression
• Linear Transformers are Versatile In-Context Learners
• Transformers Don't In-Context Learn Least Squares Regression
• Linear attention is (maybe) all you need (to understand transformer optimization)