2022/07/08 by Elias Fekhari, Bertrand Iooss, Fekhari, Elias +7
Decision Sciences · Engineering · #FOS: Mathematics #Fault Detection and Control Systems #Industrial Vision Systems and Defect Detection #Probabilistic and Robust Engineering Design #Statistics Theory (math.ST)
paper · pdf · doi:10.48550/arxiv.2207.03724
openalex publication_date 2022/07/08 · openalex created_date 2022/07/13 · openalex updated_date 2026/07/28
Unbiased assessment of the predictivity of models learnt by supervised\nmachine-learning methods requires knowledge of the learned function over a\nreserved test set (not used by the learning algorithm). The quality of the\nassessment depends, naturally, on the properties of the test set and on the\nerror statistic used to estimate the prediction error. In this work we tackle\nboth issues, proposing a new predictivity criterion that carefully weights the\nindividual observed errors to obtain a global error estimate, and using\nincremental experimental design methods to "optimally" select the test points\non which the criterion is computed. Several incremental constructions are\nstudied, including greedy-packing (coffee-house design), support points and\nkernel herding techniques. Our results show that the incremental and weighted\nversions of the latter two, based on Maximum Mean Discrepancy concepts, yield\nsuperior performance. An industrial test case provided by the historical French\nelectricity supplier (EDF) illustrates the practical relevance of the\nmethodology, indicating that it is an efficient alternative to expensive\ncross-validation techniques.\n