When and How Unlabeled Data Provably Improve In-Context Learning

2025/06/18 by Yingcong Li, Xiangyu Chang, Li, Yingcong +8 · 2 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Machine Learning and Algorithms #Machine Learning and Data Classification #Optimization and Control (math.OC)

paper · pdf · doi:10.48550/arxiv.2506.15329

openalex publication_date 2025/06/18 · openalex created_date 2025/10/19 · openalex updated_date 2026/07/28

Abstract

Recent research shows that in-context learning (ICL) can be effective even when demonstrations have missing or incorrect labels. To shed light on this capability, we examine a canonical setting where the demonstrations are drawn according to a binary Gaussian mixture model (GMM) and a certain fraction of the demonstrations have missing labels. We provide a comprehensive theoretical study to show that: (1) The loss landscape of one-layer linear attention models recover the optimal fully-supervised estimator but completely fail to exploit unlabeled data; (2) In contrast, multilayer or looped transformers can effectively leverage unlabeled data by implicitly constructing estimators of the form ∑_{i≥ 0} a_i (X^\top X)ⁱX^\top y with X and y denoting features and partially-observed labels (with missing entries set to zero). We characterize the class of polynomials that can be expressed as a function of depth and draw connections to Expectation Maximization, an iterative pseudo-labeling algorithm commonly used in semi-supervised learning. Importantly, the leading polynomial power is exponential in depth, so mild amount of depth/looping suffices. As an application of theory, we propose looping off-the-shelf tabular foundation models to enhance their semi-supervision capabilities. Extensive evaluations on real-world datasets show that our method significantly improves the semisupervised tabular learning performance over the standard single pass inference.

Citations

• MAPLE: Many-Shot Adaptive Pseudo-Labeling for In-Context Learning
• Gating is Weighting: Understanding Gated Linear Attention through In-context Learning
• TabICL: A Tabular Foundation Model for In-Context Learning on Large Data
• TabDPT: Scaling Tabular Foundation Models on Real Data
• On the Training Convergence of Transformers for In-Context Classification of Gaussian Mixtures
• In-Context Learning with Representations: Contextual Generalization of Trained Transformers
• Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters
• Fine-grained Analysis of In-context Linear Estimation: Data, Architecture, and Beyond
• A Theoretical Understanding of Self-Correction through In-context Alignment
• Many-Shot In-Context Learning
• Visual In-Context Learning for Large Vision-Language Models
• In-Context Learning with Transformers: Softmax Attention Adapts to Function Lipschitzness
• How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression?
• One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attention
• Trained Transformers Learn Linear Models In-Context
• Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selection
• Expectation Maximization Pseudo Labels
• Transformers as Algorithms: Generalization and Stability in In-context Learning
• Transformers learn in-context by gradient descent
• What learning algorithm is in-context learning? Investigations with linear models
• What Can Transformers Learn In-Context? A Case Study of Simple Function Classes
• TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second
• Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
• An Explanation of In-context Learning as Implicit Bayesian Inference
• Risk Bounds for Over-parameterized Maximum Margin Classification on\n Sub-Gaussian Mixtures
• Binary Classification of Gaussian Mixtures: Abundance of Support Vectors, Benign Overfitting and Regularization
• Theoretical Insights Into Multiclass Classification: A High-dimensional Asymptotic View
• Theoretical Analysis of Self-Training with Deep Networks on Unlabeled Data
• Language Models are Few-Shot Learners
• Finite-sample Analysis of Interpolating Linear Classifiers in the Overparameterized Regime
• A Model of Double Descent for High-dimensional Binary Linear Classification
• Asymptotic Bayes risk for Gaussian mixture in a semi-supervised setting
• Overfitting or perfect fitting? Risk bounds for classification and regression rules that interpolate
• Statistical guarantees for the EM algorithm: From population to sample-based analysis
• A Closer Look at TabPFN v2: Understanding Its Strengths and Extending Its Capabilities
• Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizers

Cited by

• In-Context Semi-Supervised Learning
• Transformers are almost optimal metalearners for linear classification

Related

• In-Context Semi-Supervised Learning
• FlatMatch: Bridging Labeled Data and Unlabeled Data with Cross-Sharpness for Semi-Supervised Learning
• AuxMix: Semi-Supervised Learning with Unconstrained Unlabeled Data
• Investigating the Learning Behaviour of In-context Learning: A Comparison with Supervised Learning
• Informative missingness and its implications in semi-supervised learning