Towards the Efficient Inference by Incorporating Automated Computational Phenotypes under Covariate Shift

2025/05/28 by Ying Chao, Jun Jin, Ying, Chao +8 · 1 citation
Biochemistry, Genetics and Molecular Biology · Computer Science · #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning and Data Classification #Methodology (stat.ME) #Single-cell and spatial transcriptomics

paper · doi:10.48550/arxiv.2505.22632

openalex publication_date 2025/05/28 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Collecting gold-standard phenotype data via manual extraction is typically labor-intensive and slow, whereas automated computational phenotypes (ACPs) offer a systematic and much faster alternative. However, simply replacing the gold-standard with ACPs, without acknowledging their differences, could lead to biased results and misleading conclusions. Motivated by the complexity of incorporating ACPs while maintaining the validity of downstream analyses, in this paper, we consider a semi-supervised learning setting that consists of both labeled data (with gold-standard) and unlabeled data (without gold-standard), under the covariate shift framework. We develop doubly robust and semiparametrically efficient estimators that leverage ACPs for general target parameters in the unlabeled and combined populations. In addition, we carefully analyze the efficiency gains achieved by incorporating ACPs, comparing scenarios with and without their inclusion. Notably, we identify that ACPs for the unlabeled data, instead of for the labeled data, drive the enhanced efficiency gains. To validate our theoretical findings, we conduct comprehensive synthetic experiments and apply our method to multiple real-world datasets, confirming the practical advantages of our approach.

Citations

• Predictions as Surrogates: Revisiting Surrogate Outcomes in the Age of AI
• On the Role of Surrogates in Conformal Inference of Individual Causal Effects
• Another look at statistical inference with machine learning-imputed data
• Valid inference for machine learning-assisted genome-wide association studies
• Can Unconfident LLM Annotations Be Used for Confident Conclusions?
• Applied Causal Inference Powered by ML and AI
• ReTaSA: A Nonparametric Functional Estimation Approach for Addressing Continuous Target Shift
• Assumption-Lean and Data-Adaptive Post-Prediction Inference
• Evaluating the Surrogate Index as a Decision-Making Tool Using 200 A/B Tests at Netflix
• Inferring the Long-Term Causal Effects of Long-Term Treatments from Short-Term Experiments
• PPI++: Efficient Prediction-Powered Inference
• Doubly Flexible Estimation under Label Shift
• Revisiting inference after prediction
• ELSA: Efficient Label Shift Adaptation through the Lens of Semiparametric Models
• Prediction-Powered Inference
• USB: A Unified Semi-supervised Learning Benchmark for Classification
• Semiparametric doubly robust targeted double machine learning: a review
• An Information-theoretical Approach to Semi-supervised Learning under Covariate-shift
• Long-term Causal Inference Under Persistent Confounding via Data Combination
• Identifiability of Label Noise Transition Matrix
• Surrogate Assisted Semi-supervised Inference for High Dimensional Risk Prediction
• Improved Estimators for Semi-supervised High-dimensional Regression Model
• Provably End-to-end Label-Noise Learning without Anchor Points
• Optimal and Safe Estimation for High-Dimensional Semi-Supervised Learning
• On the role of surrogates in the efficient estimation of treatment effects with limited outcome data
• A Unified View of Label Shift Estimation
• Visually Communicating and Teaching Intuition for Influence Functions
• Assumption Lean Regression
• Semi-supervised Inference for Explained Variance in High-dimensional\n Linear Regression and Its Applications
• Marginal Singularity, and the Benefits of Labels in Covariate-Shift
• Detecting and Correcting for Label Shift with Black Box Predictors
• Semi-Supervised linear regression
• Semi-supervised Inference: General Theory and Estimation of Means
• The Influence Function of Semiparametric Estimators
• A Computational Approach to Politeness with Application to Social\n Factors
• On Causal and Anticausal Learning
• Semi-Supervised Learning of Class Balance under Class-Prior Change by\n Distribution Matching

Cited by

• SADA: Safe and Adaptive Aggregation of Multiple Black-Box Predictions in Semi-Supervised Learning

Related

• Phenotyping with Positive Unlabelled Learning for Genome-Wide Association Studies
• Phenotype Inference with Semi-Supervised Mixed Membership Models
• Interpreting Supervised Machine Learning Inferences in Population Genomics Using Haplotype Matrix Permutations
• Computational Phenotype Discovery via Probabilistic Independence
• Efficient Inference under Label Shift in Unsupervised Domain Adaptation