On Leakage in Machine Learning Pipelines

2023/11/07 by Leonard Sasse, Sasse, Leonard, Eliana Nicolaisen‐Sobesky +24 · 1 voice · 3 citations
Computer Science · #Artificial Intelligence (cs.AI) #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #cs.AI #cs.LG

paper · pdf · doi:10.48550/arxiv.2311.04179

openalex publication_date 2023/11/07 · arxiv published 2023/11/07 · arxiv updated 2024/03/05 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/30

Abstract

Machine learning (ML) provides powerful tools for predictive modeling. ML's popularity stems from the promise of sample-level prediction with applications across a variety of fields from physics and marketing to healthcare. However, if not properly implemented and evaluated, ML pipelines may contain leakage typically resulting in overoptimistic performance estimates and failure to generalize to new data. This can have severe negative financial and societal implications. Our aim is to expand understanding associated with causes leading to leakage when designing, implementing, and evaluating ML pipelines. Illustrated by concrete examples, we provide a comprehensive overview and discussion of various types of leakage that may arise in ML pipelines.

Cited by

• A geometric and deep learning reproducible pipeline for monitoring floating anthropogenic debris in urban rivers using in situ cameras
• ML2B: Benchmarking LLMs on Cross-Lingual ML Pipeline Generation
• Outlier Detection in Plantar Pressure: Human-Centered Comparison of Statistical Parametric Mapping and Explainable Machine Learning

Discussions

• arxiv.org/abs/2311.04179 [bsky, 4 points, 0 comments]

Related

• Don't Push the Button! Exploring Data Leakage Risks in Machine Learning and Transfer Learning
• Leakage and the Reproducibility Crisis in ML-based Science
• LeakageDetector: An Open Source Data Leakage Analysis Tool in Machine Learning Pipelines
• Underspecification Presents Challenges for Credibility in Modern Machine Learning
• Survey: Leakage and Privacy at Inference Time