Localising Dropout Variance in Twin Networks

2025/07/04 by Doyle, Cooper
Computer Science · Psychology · #Adversarial Robustness in Machine Learning #Artificial Intelligence (cs.AI) #Computability, Logic, AI Algorithms #Deception detection and forensic psychology #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML)

paper · pdf · doi:10.48550/arxiv.2507.03622

openalex publication_date 2025/07/04 · openalex created_date 2025/10/20 · openalex updated_date 2026/08/01

Abstract

Accurate individual treatment-effect estimation demands not only reliable point predictions but also uncertainty measures that help practitioners locate the source of model failure. We introduce a layer-wise variance decomposition for deep twin-network models: by toggling Monte Carlo Dropout independently in the shared encoder and the outcome heads, we split total predictive variance into an encoder component (σ_enc²) and a head component (σ_head²), with σ_enc² + σ_head² ≈ σ_tot² by the law of total variance. Across three synthetic covariate-shift regimes, the encoder component dominates under distributional shift (ρ_enc=0.53) while the head component becomes informative only once encoder uncertainty is controlled. On a real-world twins cohort with induced multivariate shift, only σ_enc² spikes on out-of-distribution samples and becomes the primary error predictor (ρ_enc ≈ 0.89), while σ_head² remains flat. The decomposition adds negligible cost over standard MC Dropout and provides a practical diagnostic for deciding whether to collect more diverse covariates or more outcome data.

Citations

• From Aleatoric to Epistemic: Exploring Uncertainty Quantification Techniques in Artificial Intelligence
• Rethinking Aleatoric and Epistemic Uncertainty
• Estimating Epistemic and Aleatoric Uncertainty with a Single Model
• A Survey of Deep Causal Models and Their Industrial Applications
• Adapting Neural Networks for the Estimation of Treatment Effects
• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
• Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
• Estimation and Inference of Heterogeneous Treatment Effects using Random Forests
• Estimation and Inference of Heterogeneous Treatment Effects using Random Forests
• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
• Deep Gaussian Processes
• Bayesian Nonparametric Modeling for Causal Inference

Related

• Identifying Causal-Effect Inference Failure with Uncertainty-Aware Models
• Counterfactual Prediction with Deep Instrumental Variables Networks
• Unreliable Uncertainty Estimates with Monte Carlo Dropout
• Why have a Unified Predictive Uncertainty? Disentangling it using Deep Split Ensembles
• Notes on the Behavior of MC Dropout