Cross validation for model selection: A review with examples from ecology

2022/11/13 by Luke A. Yates, Zach Aandahl, Shane A. Richards +1 · 39 citations
Environmental Science · Computer Science · #Species Distribution and Climate Change #Data Analysis with R #Soil Geostatistics and Mapping

paper · doi:10.1002/ecm.1557

Abstract

Abstract Specifying, assessing, and selecting among candidate statistical models is fundamental to ecological research. Commonly used approaches to model selection are based on predictive scores and include information criteria such as Akaike's information criterion, and cross validation. Based on data splitting, cross validation is particularly versatile because it can be used even when it is not possible to derive a likelihood (e.g., many forms of machine learning) or count parameters precisely (e.g., mixed‐effects models). However, much of the literature on cross validation is technical and spread across statistical journals, making it difficult for ecological analysts to assess and choose among the wide range of options. Here we provide a comprehensive, accessible review that explains important—but often overlooked—technical aspects of cross validation for model selection, such as: bias correction, estimation uncertainty, choice of scores, and selection rules to mitigate overfitting. We synthesize the relevant statistical advances to make recommendations for the choice of cross‐validation technique and we present two ecological case studies to illustrate their application. In most instances, we recommend using exact or approximate leave‐one‐out cross validation to minimize bias, or otherwise k ‐fold with bias correction if k < 10. To mitigate overfitting when using cross validation, we recommend calibrated selection via our recently introduced modified one‐standard‐error rule. We advocate for the use of predictive scores in model selection across a range of typical modeling goals, such as exploration, hypothesis testing, and prediction, provided that models are specified in accordance with the stated goal. We also emphasize, as others have done, that inference on parameter estimates is biased if preceded by model selection and instead requires a carefully specified single model or further technical adjustments.

Citations

• A guide to Bayesian model selection for ecologists
• The ASA Statement on p -Values: Context, Process, and Purpose
• Model averaging in ecology: a review of Bayesian, information‐theoretic, and tactical approaches for predictive inference

Cited by

• What makes a forest growth model climate-sensitive? An examination of statistical and silvicultural model needs under climate change
• Language, economic and gender disparities widen the scientific productivity gap
• Simulated Pulses of Senescent Plant Matter Structure Brown Food Web Pathways in an Arid Ecosystem
• Multi‐response phylogenetic mixed models: concepts and application
• Effects of agricultural land conversion and interstitial sediment on Tasmanian stream macroinvertebrate communities
• Separating intrinsic from extrinsic stomatal control in different soils
• Development of new global lake brGDGT-temperature calibrations: advances, applications, challenges, and recommendations
• Using resurvey data to predict changes in ecosystem functioning across protected and unprotected coastal dunes
• A Data-Driven Framework for Digital Transformation in Smart Cities: Integrating AI, Dashboards, and IoT Readiness
• Protection efforts have resulted in ~10% of existing fish biomass on coral reefs
• Comparison of machine learning and validation methods for high-dimensional accelerometer data to detect foot lesions in dairy cattle
• Future climatically stable habitat of a vulnerable medicinal tree, Warburgia salutaris (Pepper-bark tree), overlaps with some protected areas in southern Africa
• Holdout cross-validation for large non-Gaussian covariance matrix estimation using Weingarten calculus
• The Relative Instability of Model Comparison with Cross-validation
• Predicting influenza-like illness trends based on sentinel surveillance data in China from 2011 to 2019: A modelling and comparative study 1 . [europepmc]
• Fast and Accurate Estimation of Selection Coefficients and Allele Histories from Ancient and Modern DNA. [europepmc]
• Electromyography- and Bioimpedance-Based Detection of Swallow Onset for the Control of Dysphagia Treatment. [europepmc]
• Optimizing bioprocessing efficiency with OptFed: Dynamic nonlinear modeling improves product-to-biomass yield. [europepmc]
• Biomarker discovery and development of prognostic prediction model using metabolomic panel in breast cancer patients: a hybrid methodology integrating machine learning and explainable artificial intelligence. [europepmc]
• Multipopulation Whale Optimization-Based Feature Selection Algorithm and Its Application in Human Fall Detection Using Inertial Measurement Unit Sensors. [europepmc]
• Artificial intelligence and machine learning in cell-free-DNA-based diagnostics. [europepmc]
• Tennis Timing Assessment by a Machine Learning-Based Acoustic Detection System: A Pilot Study. [europepmc]
• Toward interpretable expected goals modeling using Bayesian mixed models. [europepmc]
• Establishing the Exposure-QT Relationship During Bedaquiline Treatment Using a Time-Varying Tuberculosis-Specific Correction Factor (QTcTBT). [europepmc]
• Practical guidelines for validation of supervised machine learning models in accelerometer-based animal behaviour classification. [europepmc]
• The Temporal Response Surface: A Novel Method for the Assessment of Delayed and Time-Cumulative Aquatic Ecosystem Risk. [europepmc]
• Randomised Badger Culling Trial-no effects of widespread badger culling on tuberculosis in cattle: comment on Mills, Woodroffe and Donnelly (2024a, 2024b). [europepmc]
• Improving Bovine Brucellosis Diagnostics: Rapid, Accurate Detection via Blood Serum Infrared Spectroscopy and Machine Learning. [europepmc]
• Machine learning reveals distinct gene expression signatures across tissue states in stony coral tissue loss disease. [europepmc]
• A Data-Driven Framework for Digital Transformation in Smart Cities: Integrating AI, Dashboards, and IoT Readiness. [europepmc]
• Neural pathways to bariatric success: What explainable AI reveals that conventional fMRI methods may miss. [europepmc]
• Language, economic and gender disparities widen the scientific productivity gap. [europepmc]
• Synching with seasonality: Predicting roe deer parturition phenology across its distributional range. [europepmc]
• Hyperspectral-based classification of individual wheat plants into fine-scale reproductive stages. [europepmc]
• Optimizing Feature Selection and Machine Learning Algorithms for Early Detection of Prediabetes Risk: Comparative Study. [europepmc]
• Functional and Clinical: An Explainable Deep Learning Model for Multimodal Alzheimer's Disease Classification. [europepmc]
• Targeted blood proteome profiling using NULISAseq identifies a high-performance biomarker panel for Aβ pathology quantification and staging. [europepmc]
• Organic carbon stocks in aboveground biomass and soils in hyper-arid AlUla County, Saudi Arabia. [europepmc]
• Machine learning-based computational validation of the Addictions Neuroclinical Assessment framework in relation to hazardous drinking. [europepmc]

Related

• Cross validation for model selection: a primer with examples from ecology
• A practical guide to selecting models for exploration, inference, and prediction in ecology
• A survey of cross-validation procedures for model selection
• Assessing the Significance of Model Selection in Ecology
• Cross‐validation strategies for data with temporal, spatial, hierarchical, or phylogenetic structure