2025/12/20 by Vassilis Digalakis Jr, Digalakis, Vassilis, Jr, Christophe Pérignon +6
Decision Sciences · Computer Science · #Advanced Bandit Algorithms Research #Explainable Artificial Intelligence (XAI) #Stochastic Gradient Optimization Techniques
paper · pdf · doi:10.48550/arxiv.2512.18390
Organizations often have an incumbent predictive model in production when new data sources become available. Because historical training data lack the new features, a challenger model must be trained on a small but growing full-feature dataset. We study whether, and when, the organization should switch to the challenger. The decision is statistical and economic: the challenger's predictive performance improves as full-feature data accumulate, but repeated retraining is costly and delays benefits from deployment. We develop a framework linking learning-curve dynamics to model-switching economics. Under a standard power-law learning curve and finite data-collection horizon T, the optimal time to train and evaluate the challenger scales as T1/(1+α): learning-curve shape (through its learning speed α) is the primary theoretical determinant of when to stop experimenting; costs determine switching profitability. Even without knowing the learning curve, the operational problem is tractable: we show that any algorithm stopping on the T2/3 scale and making reliable switch/discard decisions achieves O(T2/3√(log T)) regret relative to a full-foresight oracle. We propose a sequential evaluation algorithm that uses local learning-curve trends to anticipate improvement, and test it in a real-world credit-scoring study. Even with this local approximation, the algorithm theoretically and empirically achieves near-oracle performance. It is also more stable than greedy sequential evaluation algorithms, where noisy early estimates trigger premature discarding, or simple one-shot evaluation algorithms, which work only when their fixed evaluation time matches the (unknown in practice) theoretical timing scale. Our framework offers a step toward principled model governance when new data sources require costly collection, validation, and deployment.